Challenge

Jesta Digital uses a Service Oriented Architecture for application development and deployment, and hosts a wide range of SOAP, REST and Hessian based services primarily over HTTP/S. The ESB is used to proxy calls to these services and perform header based routing, load balancing and fail-over as well as asynchronous statistics recording, logging, and error handling.

Internally, a custom built application is used to manage these services deployed on hundreds of server instances as well as the interactions between them. Due to very high time to market requirements software changes are continuously delivered and deployment changes happen frequently. Consequently the ESB outing configuration requires to be updated. The services process approximately 60 to 80 million requests a day, and require being available 24x7.

Reasons For Migration

An Open Source ESB had been in use for almost 3 years at Jesta Digital. However with time, a healthy open source community and active development around the underlying open source project reduced noticeably. New releases started getting delayed, while some open issues remained unresolved.

Persisting issues like missing native support for Hessian and REST(internally all messages were processed as SOAP) as well as side effects during frequently required configuration changes (reestablishment of persistent connections, unequal distribution across nodes, several connection errors) were unlikely to be addressed due to constraints inherent to the architecture and design based on top of an underlying web service engine.

Moreover the custom extension code developed over the ESB by Jesta Digital was becoming increasingly complex to write and maintain due to the absence of a clearly defined end-user level API.

Solution

The free and open source UltraESB was launched in early 2010 by AdroitLogic, founded by the former architect of the Apache Synapse and WSO2 ESBs, Asankha Perera. In September 2011, Ruwan Linton who initially succeeded Asankha at WSO2 as its next ESB architect, also joined AdroitLogic as its Director of Engineering. “Asankha and Ruwan were well known to us for their outstanding production support and technical prowess. The underlying technological innovations of the UltraESB coupled with its focus on simplicity of use and management, made it a strong candidate against all other options” said Eric Hubert, Executive Director, Strategy Architecture of Jesta Digital.

The similarity of the core concepts, and the visibly reduced dependence on a large and complex family of third party libraries, and a cleanly separated end-user API as a separate Maven artifact simplified the migration. AdroitLogic heavily supported the whole migration process by delivering an initial cut of all custom mediation and configuration generation code ported to the UltraESB API free of charge. Additionally AdroitLogic supported the Jesta Digital team to load test the migrated solution using a customized version of the ESB Performance Benchmark (http://esbperformance.org). The performance testing independently proved the earlier benchmark results published by AdroitLogic. Stability tests executed over multiple days under heavy load, gave evidence of more efficient resource utilization and improved stability.

Deployment

Jesta Digital runs two separate clusters of 3 nodes each, across three physical machines, and uses a hardware load balancer to distribute the load between the ESB nodes. The two clusters perform different tasks, and are independently managed. Configurations generated for routing are first checked into a version control system, a nd then tested in separate integration and staging systems. Once a change is approved and ready for live deployment, it is tagged in the version control system, promoted to the shared live storage and applied cluster-wide using a UTerm command to switch the configuration without any interruption at any node. The UTerm command line interface / scriptable client and the web based UConsole provide management and monitoring capabilities for the ESB clusters.

Each ESB instance reports its metrics via JMX, and a Zabbix server records this information for historical analysis/graphing, and to fire alert triggers on certain thresholds to notify system administrators. Message statistics are asynchronously reported to a statistics database and used by internal systems at Jesta Digital for reporting.

The main cluster of 3 nodes processes 60 to 80 million requests per day, with peaks of ~3,000 messages per second. The system operates with an extremely small memory footprint and thus hasa very low garbage collection (GC) overhead due to its use of a RAM disk and Zero-Copy proxying for performance. The average heap memory size of ~300MB causes minor collections of around ~7ms using the CMS collector. The average CPU utilization is approximately ~5%, while mediation time within the ESB is about 200 micro seconds on average.

Future

Jesta Digital has already embarked on moving more applications to the UltraESB, and is currently investigating the integration of a highly available messaging infrastructure with the UltraESB to support asynchronous and guaranteed message delivery. In addition, routing to auto-scaled service endpoints as well as improved API management are on the road map.