HOW A CANADIAN BANK REDUCES THEIR RISK EXPOSURE

The Problem  

The in-house system in a major Canadian bank responsible for preparing, processing, and aggregating cash flow data for a regulatory report hit the scalability bottleneck. The team considered the possibility of moving the ETL and the computing modules to the public cloud, but the bank security and confidentiality requirements would cause a major approval delay and present a chance of missing the deadline. Time was ticking.

  

The Initiative  

After consulting with the GoodLabs Fintech Studio team, a set of requirements were identified. The solution should work on-prem but should easily be able to be moved to the public cloud. It should be scalable as the data volume increases. The solution should also allow changes in the implementation stack.

Due to tight deadlines and a deep familiarity with the business process requirements, the team decided to take a pragmatic approach and initially use the code developed by the business team and gradually apply alterations to improve the system, but this approach should have no impact on the delivery of the system at any stage.

The Solution

  1. After reviewing all the requirements, the bank collaborated with us to design the architecture and implemented the proof of concept for validation.

  2. The team first dockerized the programs and reimplemented the ETL logic using GoLang instead of Python. Through this, the bank was able to improve the ETL performance by 39 times. 

  3. The team then introduced the micro batching and designed an end-to-end pipeline using Argo, a cloud-native workflow manager, to run the cash flow data processing on Kubernetes in parallel as the data arrived. The cash flow data was stored in the MinIO object store in Kubernetes. Originally stored as CSV files, the cash flow data grew from a daily volume from 190GB to more than 2TB. 

  4. The team then decided to change the format to parquet format and made the data available to the downstream cash flow aggregator via MinIO’s S3 layer.

The Result  

When the new micro-batch cashflow’s end-of-date processing was deployed in the production, the bank reduced the total cash flow report processing time from 5 hours to 30 minutes and dramatically reduced the risk exposure associated with late reporting.