UK retail bank supports flexible ML workloads in the cloud

The Challenge

ECS had already designed and delivered an AWS-hosted environment to replace an aging retail risk analytics platform for this leading UK retail bank. Following this, the bank now embarked on a 2‑year programme to implement a cloud‑native and enterprise‑scale data lake. ECS was engaged to deliver both the technical design and implementation.

The bank’s existing data lake was supported across Hadoop clusters out of its on-premise data centres. This presented the bank with quite constraining logistical challenges making it difficult to quickly spin up the right resources for analytic workloads, particularly in support of Machine Learning training and execution. In order to cater for this kind of fluctuating demand the Bank would have required the planning for and provisioning of sufficient resources in the data centre to be able to support peak levels of activity. This would have effectively meant that expensive resources were frequently underused or left idle. At other times the available capacity was inadequate to service peak data and analytical workloads in a timely manner. It was important to the bank that access to existing information was not interrupted as the programme progressed to build the new data lake.

The Solution

ECS appreciated that critical to the success of this project was properly understanding the types of data being handled as well as recognising the different source patterns of the data. A framework was developed that would allow ECS to define a series of different paths into the data lake applicable for the data to use each path.

Data was categorised as transactional or table-copy and could be partitioned for collation from one of three source patterns:

• data already curated in Parquet in the on-premise data lake;
• on premise golden source data, to ingest directly into cloud;
• un-curated data that is already published in the cloud.

At this point all of the data in scope for the project was structured, but the categories and patterns will apply equally for unstructured data.

Using AWS native tools, a pathway into the cloud-based data lake for each category of data from each of the sources was established. This included using AWS SFTP and AWS Kinesis to ingest data into an S3 bucket to be processed using AWS Glue ETL to transform the data into  Parquet format and stored in the data lake. AWS Glue was used to create and maintain data schemas used by data scientists to access data through AWS Athena.

With all of the design, infrastructure and cloud services in place, it was then possible to begin filling the lake in a controlled manner and supporting analytical workloads . From the outset of the project to the first production data loads being ingested into the data lake took less than 5 months.

The Benefits

• Accelerated implementation: Access to and provisioning of enterprise grade SaaS platforms and services enables increased rates of change and adoption to keep pace with competitors; zero-day provisioning of platforms and services has accelerated adoption of Artificial intelligence and machine learning which are key to strategic objectives.
• Scalability: Elastic storage and compute combine to ensure capacity to support fluctuating demand; this ability to scale on demand has decreased cost of compute and storage dramatically while increasing application performance and service, delivering increasing overall ROI and NPS score.
• Repeatability: Infrastructure As Code has enabled true DevOps practices across the business which are driving rapid, high quality solutions at speed; release cycles reduced to hours or days from weeks or months.
• Fully managed: as a Service (aaS) platforms and services, allows us to focus on Customer value add activities.
• Data driven: Data is available and consumable to support decision making for all data consumers across the organisation; removing data silos in legacy systems and co-locating data with cloud processing compute power and a variety of tools in the cloud supports a truly data driven culture; dynamic and interactive dashboards & real-time reporting with enterprise security and governance provides staff with self-service insight in familiar and easy to consume applications and tools.
• Performance: a large join query which used to take approximately 13 minutes, was reduced to 3 seconds.
• The agile approach utilised allowed the bank to fail fast, learn quickly and rapidly increase the pace of innovation.
• Significant reduction in run costs, due to the elastic scaling and pay-per-use service model.