Unified data pools in multi-national airlines and energy companies

Querying distributed, heterogeneous data sources

Numerous organizations store data in multiple systems, with databases and file systems as the most common types of storage platforms. In some cases, different departments within the same organization use different systems and technologies to store data. For example, an organization might have a data lake that contains many different types of data: databases and file systems are among the more common types of data storage platforms. These can be distributed over multiple locations, and they may also be subject to various data regulations.

Most of these databases achieve tight coupling between storage and processing engines. For instance, a DBMS typically assumes that data is already stored within the DBMS before querying it. In other words, true data independence is not yet a reality today. As a result, it is crucial to be able to run analytics over multiple processing and storage platforms. As a result, organizations are required to run analytics over data lakes in a transparent manner, i.e., without even noticing that they are querying a data lake (multiple storage platforms) using different processing platforms. The current practice today is to perform tedious, time-intensive, and costly data migration tasks as well as complex integration tasks for analyzing multiple data sets to get the best possible probability and insight.

‍

Blossom Sky's Virtual Data Lakehouse brings intelligence to any data source

The  Blossom Sky platform is designed to bring intelligence capabilities to data sources rather than the data to data warehouses or lakes. Blossom Sky hides the heterogeneity of storage and processing systems from users, who simply write their applications on top of Blossom Sky and let it take care of transparently executing such applications over data lakes: taking care of any required data movement and transformation. Blossom Sky shields users from all these tedious tasks, allowing them to instead focus on the logic of their data analytics. We showcase how Blossom Sky operates on different data lakes in different locations in Geo Exploration and Airline Management.

‍

Geo Exploration

A typical energy company produces more than 1.5 TB of diverse datasets per day, most of which are structured and semi-structured. These data come from heterogeneous sources, such as sensors, GPS devices, drilling sensors, geothermal sensors, transportation, tanks, ships and other edge driven instruments and sources. For example, during the exploration phase for a reservoir that might be profitable to drill, geologists and geophysicists must acquire, integrate and analyze data in real time to predict if the reservoir would be profitable based on the physical properties of rocks. They must remove noise from real-time seismic data coming from downhole sensors in exploratory wells producing oil or gas; integrate the cleaned sensor data with historical drilling and production data; visualize volume and surface renderings to formulate hypotheses and verify them with ML methods such as regression and classification using emails and reports filed in cabinets if they exist.

The dataflow shows the components necessary for an overview of Blossom's solution, which helps energy companies achieve sustainability and environmental stability by exploring fossil resources.

‍

‍

Airline Management

Before commercial airplanes can take off, a series of systems must work together to coordinate flight operations. In more detail: several weeks before departure, passenger booking systems produce daily forecasts for expected passenger load and baggage weight. These predictions are then consumed by cargo systems to begin accepting cargo loads. Few days before departure, crew scheduling systems assign staff for the flight. The engineering system is highly instrumented and produces large amounts of sensor data: they especially look for outliers to carry out pre-emptive and predictive maintenance. Similarly, catering systems plan food preparation based on the predicted number of passengers.

When a flight takes off, the aircraft is weighed and its cargo is counted. Data on these figures is stored on an historical system. Some years ago, the datasets were much smaller than they are today. Airlines are always under pressure to operate time efficiently—on the best fuel efficiency and need to be managed on the highest optimal level to mitigate risks (as we had and have during the pandemic). The Dow Jones Sustainability Index displays the best data driven and sustainable airlines, and they all have something in common - they use data as an asset. The next picture shows how a typical data flow for such an extraordinary airline using Blossom Sky looks like in a high level overview.

‍

‍

Conclusion

Federated data lakes are a powerful technique that allows for the sharing and integration of data from multiple sources, such as data silos and data lakes, while still maintaining the security and privacy of the data. This is achieved by training models on decentralized data without the need to transfer or consolidate the data in a central location. This approach can be especially useful for organizations that are dealing with sensitive data and must comply with regulations such as HIPAA and GDPR.

Implementing a Virtual Data Lakehouse helps organizations combine data from multiple sources, such as data silos and data lakes, without having to move or consolidate the data, while still being able to build accurate models and improve decision-making. Additionally, by using federated data lakes, organizations can ensure compliance with data regulations by keeping the data within the secure environments where it was originally collected and by using techniques such as data de-identification, encryption, and access control.

Overall, the virtual data lakehouse has been proven to be an effective solution to the challenges of data silos and data regulations by allowing organizations to share and integrate data from multiple sources while maintaining data security and privacy. Blossom Sky is the leading virtual data lakehouse for federated data lakes, and it optimizes the business value of data at scale. The platform enables big data analytics and artificial intelligence by implementing a groundbreaking way to operate petabytes of data across multiple data silos and data lakes. Blossom Sky does not rely on any centralized knowledge for decentralized analytics; it empowers your employees to run data analytics and AI tasks directly where the data lives.

Data mesh and data platform abstraction are not silver bullets or one-size-fits-all solutions. They require careful planning, design, implementation, and governance. They also require a cultural shift from centralized to decentralized data ownership and collaboration. DataBloom's Virtual Data Lakehouse offers a promising vision for how organizations can harness the power of data to deliver better value for their providers, partners, and stakeholders. Be sure you undergo a brief consultation with your DataBloom AI representative to address the challenges of implementing Blossom Sky into your data strategies.

‍

Research reference:
Prescriptive Learning for Air-Cargo Revenue Management (under participation of Walmart Global Tech)