Approach and challenges of Open Banking APIs with CQRS and Core Banking Mainframe
Tony Kambourakis, Principal Architect, IBM
Sangeetha Sridharan, Information Architect, IBM
Simon Hudson, Senior Managing Consultant, IBM
Overview
In supporting Open Banking an organisation may wish to make data managed by their core banking system (typically running on a mainframe) and customer Master Data Management (MDM) available for retrieval via the Open Banking API. They may have an existing API that retrieves this data directly from these systems for their Internet Banking application. In some cases, the banking organisation may have accumulated multiple brands that result in many core systems resulting in multiple source systems.
One approach is to establish a capability to extract the data from a banking organisation’s core systems into a separate, read-optimised data store that could be used to expose data to their Open Banking API and Internet Banking application.
The goals include:
- Reducing on-going mainframe costs by minimising the direct consumption of mainframe services from Internet Banking and Open Banking applications.
- Consolidate data from various core systems accumulated through years of brand acquisitions into a single data layer to retrieve from.
In this article we look to explore a solution, highlighting some of the key challenges along the way.
Solution
The solution comprises a read-optimised data store to retain a copy of the data that includes account, transaction and customer information. The data store receives the feed of data from source systems via Kafka. CDC is used to detect and publish changes to the source system databases and publish the changes as events into Kafka. CDC publishes the events in a raw data format that mirrors the schema of the databases. The data is transformed, via a Stream Processor System, into a canonical format to persist the data in a read-optimised form in the data stores and to drive the publishing of business events that supports an event-driven architecture within the enterprise.
The data store is divided into separate databases, one for each domain that are accessible via an associated domain microservice. The microservices provide an API that is based on the canonical model and exposes the APIs across external channels and internal applications.
Using CDC to publish raw events
The underlying event driven architecture of the system establishes a set of business events that are produced and consumed by various systems as part of a choreography of events. The events present business significant occurrences such an account being created (account origination), a customer’s details being updated, a financial transaction occurring and a scheduled payment being created.
One approach is to have the source systems of these events publish them directly to the event/messaging system. However, this requires the source system to either already support this mechanism or to be modified to publish events. Change Data Capture (CDC) provides a non-intrusive way of detecting the changes within the source system and publish the changes as events.
This approach leads to the technical challenge where the events published by the CDC system are a direct reflection of the source system’s internal database schema.
The events published by the CDC system are fine grained and will likely not include the necessary data to build an associated business event. As such, an involved series of transformation and enrichment steps are needed to combine and form the data required in the business event. Some data is spread across multiple tables in the source system database. CDC will publish the changes to each table as separate events.
To build a single business event multiple raw events would need to be combined requiring a window-based process within the Stream Processing System. That is, when a raw message is received, the Stream Processing System will allow for a certain window of time to receive subsequent events to combine the data into the single business event. In some cases, the changes to the tables do not include all the data that would be required for the business event and so an Intermediate Data Cache will need to be maintained to add the data necessary to complete the business events schema.
The Intermediate Data Cache is maintained as a replica of the subset of tables required when CDC publishes raw messages to the kafka topics. The enrichment of meaningful business event is performed by combining the source raw message with the data in the Intermediate Data Cache.
The events were identified where more than one underlying source table gets changed, these events combined the data stream from multiple topics (stream join) and published as events.
In converting a raw event to an associated business event, the changes applied on the raw table columns are identified as CDC includes the before and after state. Though in some cases it is not possible to identify the raw event as the various combinations of arrival times and content may be too complex. To work around that the events can be grouped by CDC and CDC may perform a back-query to the source system before publishing the raw event in order to enrich the raw event with the required additional column information. This helps avoid ambiguous event creation and reducing the use of the Intermediary Data Cache. There is an impact to performing the back-query against the source system database but the core system processing component is not impacted and so this approach is used sparingly.
The event schemas are also standardised to make avoid source system specific values. For instance, an active account status could be stored with a value of “O” (representing open) or “A” (for Active) or “1” (true ) in that specific source system. To make this as a system agnostic representation a standardization representation may be introduced. This was achieved using a reference look up, which acts as a conversion mechanism from source specific values to standard values when data being persisted and published. This reference changes from source are periodically updated from the source master data management system.
Additional areas that will have further challenges include identifying missing data, capturing what cannot be published as a successful event and reprocessing events to keep the data integrity. Some scenarios that may need to be handled include late arriving stream events from source systems, late arriving updates to intermediary cache, missing data in reference lookup, race issue with events occurring at the same time with few seconds/ microseconds difference, batch processing resulting in multiple commits on the same record.
Event Granularity
The granularity and content of business events are also a challenge with striking the right balance. Course grained events such as Account Created and Account Updated will include a large set of data that reflects various attributes relevant for the event.
The Account Updated event contains both attributes that have been changed along with attributes that have not been changed. This is to provide the contextual details of the account that have been updated to consumers without requiring the consumer to make a subsequent call to retrieve the details of the account that have not been changed. Conversely a fined grained event will focus on the specific change, in the case of the Account Update finer grained events could include Account Status Updated, Account Fraud Status Updated, Account Pricing Group Updated, Account Balance Updated.
Whilst the finer grained events are clearer in their change consumers are subscribing to far more events of which the order may be critically important. One approach is to keep the course grained event of Account Updated but include a “change set” attribute that indicates which related data block within the structure has been updated to help the consumer narrow down on the nature of the change that drove the event. The Account Balance event could be a deviation from this approach where the Account Update events are significantly driven by balance changes. For consumers that are not interested in balance changes this adds the burden of filtering out most of these events on the consumer. As such a separate Account Balance event can be defined to avoid this burden.
The business event is used as both an event that will underpin an event-driven architecture within the enterprise, allowing for both interface and temporal decoupling between systems, and the means to essentially copy the source system data to the read-optimised store.
Event Ordering
Strict ordering is required at various stages right from CDC, Kafka and the Stream Processing System until the data is persisted to the data store. Identifying the right partition key for each domain and event is a key factor in determining and maintaining the order for publishing events. The audit columns of source tables that include timestamps can be included in the raw events from CDC and may also be used to determine the order of events. This helps to maintain idempotency when the same records are reprocessed at the consumer level. These aspects must be considered by the Stream Processing System to ensure processing and writing to the database is made in strict order of the events.
Core System Logic
Not all account data can be stored in the read-optimised data store; some required calculations (certain account balances) and therefore the account microservice would make API calls to mainframe whilst other types would be retrieved from data store.
An example of a calculated field may be available balance on an account. With data such as available balance, there are two reasons not to persist this in the read only database layer:
- consumers need to be 100% confident that the data is accurate at a point in time — eventual consistency may not be enough for consumers of this field
- “available balance” may not be persisted in the core banking DB, often it is derived each time it is needed — therefore the same logic that is used to calculate it in the core would need to be replicated in the Data Services layer
To solve for this use case, APIs exposed by the Core Banking Platform can be used to augment the data that exists in the read-optimised data store.’
Other use cases for calculated fields that may follow a similar pattern include redraw amounts (min/max), and interest rates on an account.
