Fortune 500 Insurance Client
Data Engineering | Cloud Transformation | Full-Stack Development
The Challenge
Facing increased demands for reporting while assuring compliance and auditability with data protection regulations such as GDPR and CCPA, their tooling was no longer able to perform critical business functions. Ocelot Consulting was engaged to modernize the environment so the current needs for reporting and fiscal closures could be met while paving the way for more sustainable & advanced analytics.
The Solution
Ocelot partnered with the client to take a modern “big data in the cloud” approach:
- Began converting ETL and enrichment functions from a custom .NET application to Cloudera Spark jobs, and guided the organization away from CSVs and row-based systems and toward Parquet and column-based systems
- Created an AWS S3 data lake and developed a custom API for it to enable role-based access control (RBAC) and auditing, and integrated the RBAC API with the client’s Active Directory (AD) system using ServiceNow for rapid and logged approvals
- Guided the organization away from Excel reporting and toward systems such as Tableau and Jupyter, and developed a Java Database Connectivity (JDBC) driver to provide RBAC for fine-grained access to the data lake
Successful Results
The organization is radically more empowered to generate current and future reports using current-standard enterprise and open-source tooling while serving as a lighthouse for other data & analytics approaches.
Technologies Used
- AWS
- Terraform
- Cloudera Hadoop
- Cloudera Spark
- AWS S3
- Presto
- Apache Parquet
- AWS Glue
- AWS Lambda
- Python
- Node.js
- .NET
- MS Active Directory
- ServiceNow