Automated Data Ingestion Process for Real-Time Insights using Azure Data Stack

Location: United States, California

Industry: Energy / Renewable Energy

Business Functions:

  • Business Development
  • Engineering
  • Analysis
  • Information Technology
  • Finance
  • Operations
  • Quality Assurance
  • Support

Problems:

Being the market leader, Opinion Dynamics provides accurate and hands-on information to its clients (energy companies) to ensure cost reduction and mitigate the use of fuel alternatives. OD was facing several following problems:

  • Foreseeing the impact of short term asset values on the grid station
  • Detecting the possible outage via smart meter events
  • Finding the outages in specific areas
  • Identification of the peak time

Solutions

Allied helped the client with data management and data engineering services as solution to their issues. Specific solutions were as follows:

  • Azure data factory to fetch the data from the energy industries using web services and ShareFile
  • Big data solution on Azure data bricks using pyspark
  • Used Machine learning algorithms like linear regression to forecast the values and trends of energy consumption analysis
  • Analyzed more than the 300GB dataset in the energy sector
  • Execution of parallel threading on the data science projects via Python Code
  • Power App integration with Power BI to get the input on run time from the user
  • Geocodio API to get US census data and analyze the areas specific energy consumption
  • Developed the azure function script to fetch the run time data from Geocodio API according to user requirements
  • Embedded, deployed and generated the power BI reports on WordPress and asp.net site.

Results / Benefits:

  • Accurate and hands-on information regarding energy and its consumption
  • Benefits on analysis on Charging and grid stations like 28% of EV owners are “overnight chargers”, 52% are low-level chargers, etc.
  • Data engineering process to reduce energy costs and mitigate the use of alternative fuels
  • Billing cost reduced up to 40% using different technology on Azure or open source

Tech Stack:

  • Azure Data Factory
  • PySpark
  • Azure data lake
  • Dask – for parallel processing
  • Scikit-Learn
  • Power BI
  • Power App
  • Python
  • Azure Data Bricks
  • Numpy
  • Asp.net Core
  • Pandas
  • Azure Functions
  • Power BI embedded

Get In Touch With Us

we believe in  cutting  edge solutions  and are committed to your  success

Let's Work Together