Fortune 500 Food Consumer Products

Data Engineering | Cloud Transformation

The Challenge

The client needed IoT sensor data from manufacturing facilities in near real-time to improve cloud analytics, business operations, and decision making.

The Solution

Ocelot was brought in to create a data pipeline that can make roughly 2 billion manufacturing data points per day available for cloud analytics in near real-time (~15 min.) of IoT tag identification and setup at the factory:

  • Standardized available data using Databricks Delta tables
  • Automated data ingestion with little to no required configuration
  • Enhanced the ability to pull ingested data using analytics and data science tools
  • Focused on pipeline sustainability and low maintenance cost: centralized logging, automated pipeline QA/QC alerting, automated unit testing
  • Built MVP for a real-time streaming solution that will be leveraged in future as factories bring new IoT devices online

Successful Results

The automated ingestion solution was used by data scientists, data analysts, and factory staff to empower analytics work. Unlocked business value included:

- Rapid new data availability: previously two weeks, now same day

- Rapid Power BI data loading: previously hours, now minutes

- $20M of projected ROI from analytics running in production

Technologies Used

  • Databricks
  • Azure DevOps CI/CD Pipelines
  • Inmation IoT Server
  • Python Spark
  • Azure Monitor
  • Kafka