RU

ETL process development and data modeling for REsurity

  • Reduced-costs-for-AWS-services

    Reduced expenses on AWS services

  • Reduced-time-for-data-uploading

    Reduced time for data uploading

  • Improved-data-quality-and-accuracy

    Improved data quality and accuracy

  • Increased-trust-and-industry-development

    Increased trust and industry development

Project overview

Within the cooperation with DICEUS, REsurety wants to update an existing system for downloading constantly updating various energy data sets. The collaboration with REsurity started in June 2021. Since then, we've already successfully finished a pilot project and several more significant projects related to a data warehouse. Our Python data engineer demonstrated his expertise by joining the project to create a new functionality module for REsurity. Now, he is working on data modeling and ETL process development for data analytics purposes.

Client information

REsurety is the leading analytics company supporting the clean energy economy. Operating at the intersection of weather, power markets, and financial modeling, they empower the industry’s key decision-makers with best-in-class value, risk intelligence, and tools to act on it.

Business challenge

The client’s fundamental need is to extract big data sets from various data sources. The data should be structured and prepared for data science analysts for further processing: risk management, and project portfolio to work effectively with customer-related projects. Big data is updated quickly (some can be changed every minute). For terabytes of data to be outputted to the required destinations efficiently and in the format needed by the data scientists’ team, data pipelines for each new energy source should be created in a respective convenient format. For example, a winter energy company is considered a single data source that needs data in a required format.

Technical challenges

The development of the ETL process is a challenging task. The key technical challenges our team faced were unstable data sources, many variations of how these data sets should be loaded, continuous data flow, and the speed and frequency of data updates. In addition, sometimes data analysts create different requirements and requests, which can slow the process.

Solution delivered

Based on general classes and methods, our developer developed and continues developing the ETL processes from scratch for each database and data set. The project’s end goal is to deliver the software product running on AWS to gather from a given data source and output the Snowflake tables.

img 1-2
Claim a 30-minute talk with our experts and get a step-by-step strategy for your project for free!
Talk to our strategist

Key features

img 2-1

  • Structured and consistent data

     With effective ETL pipelines, data scientists have access to structured data in the convenient format required by the customer-related project.

  • Data formats suitable for data science

     According to the data analysts’ requirements, data is extracted, processed, and uploaded to the respective source in a convenient format.

  • Continuous data flow

     Our team ensured the appropriate management of continuous data flow to timely deliver the needed data to the Snowflake tables.

Value to our client

  • Reduced expenses on AWS services

    Ineffective ETL pipelines run on AWS can cost lots of money to the client. Thus, data should be effectively transformed into the Snowflake tables in a structured format.

  • Reduced time for data uploading

    Data modeling should be done for data sets that can range up to 10 000 000 items and more to effectively extract it, with no time delays when data scientists load the data.

  • Improved data quality and accuracy

    ETL pipelines should be stable and reliable. It allows data analysts to get timely, high-quality data in the required and correct format.

  • Increased trust and industry development

    Due to the robust SaaS solutions that REsurity provides to its customers, the renewable energy companies, the energy industry can develop more efficiently.

Our tech stack

  • python Python
  • SQL SQL
  • snowflake Snowflake
  • aws-1 AWS
  • bitbucket Bitbucket Pipelines

Software solutions bringing business values

gartner
5/5
6 reviews
clutch
4.9/5
48 reviews

    Contact us

    100% data privacy guarantee

    Thank you!
    Your request has been sent
    We will get back to you as soon as possible

    USA (Headquarters)

    +16469803276 2810 N Church St, Ste 94987, Wilmington, Delaware 19802-4447

    Denmark

    +4531562900 Copenhagen, 2900 Hellerup, Tuborg Havnepark 7

    Poland

    +48789743438 ul. Księcia Witolda, nr 49, lok. 15,
    50-202 Wrocław

    Lithuania

    +4366475535405 Vilnius, LT-09308,
    Konstitucijos ave.7
    6th floor

    Faroe Islands

    +298201515 Smærugøta 9A, FO-100 Tórshavn,
    Faroe Islands

    Austria

    +4366475535405 Donau-City-Straße 11 - Ares Tower, 1220 Wien

    UAE

    +4366475535405 Emarat Atrium, 423 Al Wasl Area, Dubai, P.O. Box 112344

    Ukraine

    +4366475535405 Vatslava Havela Boulevard, 4,
    Kyiv
    Case study
    ETL process development and data modeling for REsurity
    diceus.com
    Case study
    ETL process development and data modeling for REsurity
    Project overview
    Within the cooperation with DICEUS, REsurety wants to update an existing system for downloading constantly updating various energy data sets. The collaboration with REsurity started in June 2021. Since then, we've already successfully finished a pilot project and several more significant projects related to a data warehouse. Our Python data engineer demonstrated his expertise by joining the project to create a new functionality module for REsurity. Now, he is working on data modeling and ETL process development for data analytics purposes.
    Client information

    REsurety is the leading analytics company supporting the clean energy economy. Operating at the intersection of weather, power markets, and financial modeling, they empower the industry’s key decision-makers with best-in-class value, risk intelligence, and tools to act on it.

    Information
    Business challenge

    The client’s fundamental need is to extract big data sets from various data sources. The data should be structured and prepared for data science analysts for further processing: risk management, and project portfolio to work effectively with customer-related projects. Big data is updated quickly (some can be changed every minute). For terabytes of data to be outputted to the required destinations efficiently and in the format needed by the data scientists’ team, data pipelines for each new energy source should be created in a respective convenient format. For example, a winter energy company is considered a single data source that needs data in a required format.

    Technical challenges

    The development of the ETL process is a challenging task. The key technical challenges our team faced were unstable data sources, many variations of how these data sets should be loaded, continuous data flow, and the speed and frequency of data updates. In addition, sometimes data analysts create different requirements and requests, which can slow the process.

    Solution delivered

    Based on general classes and methods, our developer developed and continues developing the ETL processes from scratch for each database and data set. The project’s end goal is to deliver the software product running on AWS to gather from a given data source and output the Snowflake tables.

    Key features
    • Structured and consistent data

       With effective ETL pipelines, data scientists have access to structured data in the convenient format required by the customer-related project.

    • Data formats suitable for data science

       According to the data analysts’ requirements, data is extracted, processed, and uploaded to the respective source in a convenient format.

    • Continuous data flow

       Our team ensured the appropriate management of continuous data flow to timely deliver the needed data to the Snowflake tables.

    Value to our client
    • Reduced expenses on AWS services

      Ineffective ETL pipelines run on AWS can cost lots of money to the client. Thus, data should be effectively transformed into the Snowflake tables in a structured format.

    • Reduced time for data uploading

      Data modeling should be done for data sets that can range up to 10 000 000 items and more to effectively extract it, with no time delays when data scientists load the data.

    • Improved data quality and accuracy

      ETL pipelines should be stable and reliable. It allows data analysts to get timely, high-quality data in the required and correct format.

    • Increased trust and industry development

      Due to the robust SaaS solutions that REsurity provides to its customers, the renewable energy companies, the energy industry can develop more efficiently.

    Tech stack used in project
    • Python
    • SQL
    • Snowflake
    • AWS
    • Bitbucket Pipelines
    diceus.com