IHS Markit is a leading global provider of information, analytics and data solutions

The company's energy customers, especially in the oil and gas industry, are constantly looking for quicker and more efficient ways to extract and leverage actionable insights from large volumes of data to help improve decision making.

IHS Markit chose EPAM—an expert in cloud advisory, data application modernization and software engineering—to build an actionable data delivery and analytics platform on Amazon Web Services (AWS). Together, IHS Markit and EPAM created a cloud-based data solution that enabled its customers to make more informed decisions using fewer resources.

Key Challenges

• IHS Markit continually sources, enhances and curates major oil, gas and energy data to provide its customers with rich insights. Those customers required a platform that could deliver immediate, robust data with responsive analytics to help them make critical business decisions—for example, where to shift resources, how to prioritize wells and acreage and to predict financial performance.
• To meet customer demands, IHS Markit needed to update its current data platform into a cloud-based data and analytics solution that could provide steady-state analytics and disruption forecasts in as close to real time as possible. The new platform also required more efficient dashboards that could handle higher volumes of data than its existing solution.

Solution Highlights

EPAM worked with IHS Markit to reimagine its energy information pipeline so that it could support large data sets and enable cross-functional team insights. By leveraging the data warehousing capabilities of Snowflake and the GPU-accelerated analytics of OmniSci, EPAM helped deliver a next-gen data analysis and integrated analytics product, as well as a future-ready mega-data integration environment. This data and analytics platform was designed to support 100 concurrent users (active queries). Query response time has improved, with hot-load response times of most queries occurring in under 100 milliseconds. 

The Results

• Transformed complex and difficult-to-maintain economic calculations, such as capital expenditure and break-evens, into cloud-based phases with configurable parameters. These extract, transform, and load (ETL) steps and the monolithic code were migrated toward serverless microservices with AWS Lambda and AWS Fargate. Each new phase of the pipeline was orchestrated and sequenced with AWS Step Functions.
• Enabled large external files, like gridded structural maps and configurations, to be stored in Amazon S3 buckets for cross-container usage.
• Orchestrated OmniSci GPU database containers through Amazon Elastic Container Service (Amazon ECS) to display analytics on millions of data sets with server-side rendering. These containers were connected to shared data files via Amazon FSx for Lustre to provide high performance for queries not stored in memory.
• Provided a highly visual, responsive and innovative product design.
• Integrated multiple data sources from various oil and gas-specific structured and unstructured data types.

Technologies Used

• HEAR FROM THE CUSTOMER

Ali Sangster
VP Analytics Strategy, IHS Markit