SGX utilizes AdroitLogic UltraESB and Enterprise Middleware Framework (EMW) to process trade and price messages with guaranteed delivery
Download PDFBackground
Singapore Exchange Limited1 (SGX) is Asia's most international stock exchange, and together with its affiliates, offers a fully integrated value chain from trading and clearing to settlement and depository services. SGX, together with its affiliates, is Asia's leading counterparty for the clearing of financial and commodity products and serves customers all over the world through multiple locations.
Headquartered in Singapore, SGX operates a derivatives sales office in Hong Kong, a liaison office in Mumbai and representative offices in Beijing, London and Tokyo. SGX is also a member of the World Federation of Exchanges and the Asian and Oceanian Stock Exchanges Federation.
Challenge
As part of SGX’s trade and price information distribution process, it was required to integrate a set of upstream systems that provide trade and price information with multiple downstream systems that consume such data. A critical component of this entire architecture was an Enterprise Service Bus (ESB) that seamlessly integrates these multiple enterprise systems to receive the trade and price information from the upstream systems and deliver these in parallel to multiple downstream systems after executing data format validations and transformations.
The overall solution required integration over multiple transports and protocols, using different transformed payloads, and had to ensure the guaranteed and in-order delivery of each message to the respective downstream systems and be highly available while being deployed over multiple data centers. It was also required to monitor and track the status of each transaction through a customized dashboard, which also provides management facilities for messages such as redelivery and archiving.
Solution
SGX selected the AdroitLogic UltraESB as the Enterprise Service Bus (ESB) of their choice, and engaged AdroitLogic to design, build, test and deploy the solution based on the UltraESB. The solution, which was built on top of the AdroitLogic Enterprise Middleware Framework’s (EMW) Publisher-Subscriber model, integrated two main upstream systems and multiple downstream systems, with retry capabilities to handle possible failure scenarios.
Messages with XML payloads are received through IBM MQ and will be dispatched to an EMW interface based on a specific transport header value. First the message payload will be validated against a specific XSD schema, and then it will be dispatched into a set of EMW subroutes that transform the message payload into several formats such as CSV, using the AdroitLogic Data Mapping Framework. After the transformations, the transformed messages will be delivered to a set of configured downstream systems such as IBM MQ queues, SFTP servers and a MS SQL Server Database.
While the main message flow uses IBM MQ for intermediate storage purposes, the ESB persists a copy of each received, processed and delivered message with its status and meta information into the ESB database for monitoring and management purposes. This database persistence is performed asynchronously, and independently of the primary message flow processing, to prevent any impact on performance. In case of a message delivery failure, the EMW Framework automatically detects the failure and retries the delivery up to a configurable number (3) of times.
The solution also includes a custom dashboard integrated with AdroitLogic Integration Monitor to manage and monitor the message flows with ease. The dashboard provides facilities to track the status of each individual message while also providing means for management operations such as manual redelivery of messages, archiving of messages and even provisioning new downstream systems or disabling any of the existing downstream systems on-the-fly.
"The use of UltraESB, along with Enterprise Middleware Framework (EMW), helped us to integrate multiple sub systems of TITAN-OTC project, while achieving the required performance, reliability and guaranteed delivery for messages. Further, we have the convenience of monitoring, auditing and debugging the lifecycle of each message through a unified interface provided by the EMW dashboard."
Ang Kian Ann
Assistant Vice President - Enterprise Services
Singapore Exchange Limited
Figure 1 : EMW Framework : Publisher-Subscriber Messaging Pattern
Business Benefits
Using UltraESB and EMW Framework, SGX was able to separate the integration logic from the business oriented systems. The deployed solution is currently utilized to process over 20,000 price and trade messages daily (each with a payload around 20K), while preserving the guaranteed and in-order delivery of the messages with automated reprocessing and redelivery on failure. The custom dashboard along with the AdroitLogic Integration Monitor, provides monitoring of the message flows, auditing, retrying message delivery and provisioning & disabling downstream systems on the fly.
Deployment
The TITAN-OTC project which is developed on the Publisher-Subscriber model of the EMW framework, is deployed on a two node UltraESB cluster which is configured to operate in active-passive mode. The active node is deployed in the Primary Data Center (PDC), and the passive node is deployed in the Secondary Data Center (SDC), while maintaining the guarantee of giving priority to the PDC whenever it is available.
In addition to that, the common portion of the EMW Framework that takes care of the database persistence of messages and audit logs is deployed as a parallel but separate project on a similar active-passive setup.
Figure 2 : TiTAN-OTC deployment architecture
1. SGX and its affiliates make no representation, warranty, guarantee or endorsement whatsoever (whether express or implied) regarding, and hereby expressly exclude all responsibility and liability for, any and all products, services, processes, solutions and/or other information set out herein or in any ancillary documentation.