Fortune 500 Healthcare Client
Cloud Transformation | Data Engineering | Full-Stack Development
The Challenge
This client’s Enterprise Data Warehouse (EDW) deployment was found lacking in performance and flexibility while high in cost. The organization had ambitions of modernization including the separation of storage and compute concerns with an elastic source of truth to empower business intelligence and analytic workloads.
The Solution
Operating across multiple workstreams with mixed client and professional services teams, Ocelot worked with the client’s information security office to create its first secured production AWS account. We then co-developed a serverless and event-driven Audit, Balance, & Control (ABaC) system for ETL from the EDW, and integrated the ETL process with a custom-developed metadata storage solution for data discovery and auditability. All of the contents of the EDW were then migrated into the data lake using a mixture of batch and Change Data Capture (CDC).
Then, an event-driven and container-based “materialized view” ETL of the data lake into AWS Redshift for scaled BI, and a custom portal to empower BI use and consumption (proxying, notifications, file sharing) were developed. The final step was to create the world’s first customer-managed MicroStrategy deployment on AWS EC2.
Successful Results
This combination of technologies has created a more scalable and performant cloud-based data and BI environment which returns insights more quickly without BI and analytics workloads bringing the EDW to its knees.
Technologies Used
- AWS
- Terraform
- AWS CloudWatch Logs
- AWS S3 AWS Lambda
- AWS Step Functions
- AWS SNS
- AWS DynamoDB
- AWS Redshift
- AWS Fargate
- Apache Parquet
- AWS Kinesis
- Python
- Node.js
- Golang
- MicroStrategy