Spark streaming big data project for Naya Technologies
Project overview
In the era of all-consuming digitalization, some extremely responsible tasks are efficiently handled by specialized software solutions. In modern planes, systems and devices are seamlessly monitored and managed by a centralized software platform. Even the black box must be integrated with a stable connected solution that would reliably gather, store, process, and immediately transmit the onboard-state information in the case of an emergency. And this is exactly the kind of solution we had the pleasure to work on for Naya.
Client information
Naya Technologies is a market-leading global provider of data platform consulting and managed development services. The company has been operating in the market since 2009, offering a wide range of services and solutions, including data science consulting, Big Data and DB-related solutions, software development, data engineering, cloud and BI solutions, and many more. Throughout its practice, the company has been powering up data management and software systems for large enterprises and leading startups.
Business challenge
A plane’s black box is a critically-important source of emergency information used to investigate flight crashes, failures, in-air accidents, and other emergencies. The versatility, performance, and reliability of the black box software is the backbone of a successful investigation. Modern devices need advanced data collection and processing powers to keep everything in check. That’s why such projects are of the utmost importance both to us and our clients.
Technical challenges
In a nutshell, we had to walk our client’s team through the specifics and properties of Spark Streaming and Kafka, which should work jointly in the client’s black box data processing solution. The black box records all sorts of plane functionality and state events and transfers them to the Kafka-powered data processing engine. Spark Streaming then collects all the events from the Kafka engine in micro-batches. The main challenge was to optimize the software to gather and process more than 300 types of events. Each event requires adapted business logic to process all the data (enrich, calculate, analyze, group data assets).
Solution delivered
The final solution that we helped build and optimize is a Spark Streaming software engine that works jointly with Kafka to efficiently process the black box data. The solution is a real powerhouse that collects and records events to subsequently extract all the detailed info from them in segments. Say, upon receiving the “speed” event, the software converts all the info into structured plane speed properties, such as its constant speed, accelerations, and braking with precise time indication. All this valuable information is then rapidly transferred to the processing engine that further structure and organizes all the received data assets.
Let’s discuss how we can help with your project
Key features
Centralized events
The software collects data on the state of engines, fuel, internal environment, and hundreds of other plane indicators.
Versatile processing
The engine provides over 300 types of processing methods for over 300 different types of events.
Real-time storing
All the events and the underlying data are collected and sent for processing in real-time, with zero downtime.
Centralized UI
The software solution is conveniently accessed and managed by a single remotely enabled UI through immediate connection.
Value to our client
Our tech stack
Python
