A large transformation program of this scale can span over multiple years, so it’s important to create incremental value and build continuous momentum. Based on our experience helping other clients with these types of transformation initiatives, our integrated team of consultants, designers and engineers put together a holistic approach that consisted of establishing a solid technology foundation, operating model and scalable data platform. First, we worked together to create key technology initiatives that would deliver on the core principles the company set out to achieve, developed maturity roadmaps and clearly defined the path to progress to the next stage. This helped ensure we could deliver business value at each step in the process.

Our client needed an end-to-end data platform, built on its existing Microsoft Azure cloud infrastructure, to unlock the full potential of its data and make it more visible. To deliver the flexible, open and scalable platform the company needed, EPAM recommended Databricks Lakehouse as the technology platform to bring all the data together and create a single version of the truth. Our team helped create a common data language for all entities within the Lakehouse (internally referred to as a common data model) to ensure proper data governance.

We leveraged Microsoft Purview as the enterprise data governance tool with the goal of integrating with Databricks Unity in the future. Additionally, open source tooling and open formats like Delta Lake gave our client the ability to increase the value of its data by allowing it to be leveraged outside of its primary use.

To help employees and analysts adopt the new platform, we knew we needed to use SQL as it was the tool of choice. We integrated Databricks SQL so teams could use the tool in exploratory environments. This allowed the company to have one version of the truth, minimize movement of data and monitor data usage.

Our client needed to have the agility and flexibility to seamlessly shift its operating model based on the product engineering teams the company works with. For example, during large enterprise projects, there is a core team that supports the platform and then EDP scrum teams are embedded within the product engineering organization to reduce communication barriers. For smaller projects, or where the teams are leveraging prebuilt capabilities, the company embeds a single person into a scrum team to accelerate the adoption of the data platform. One of our goals was to provide an operating system for end-to-end data product development, which includes areas such as MLOps and DevSecOps. Working together, we helped establish a successful process to onboard and work with data product teams, so the company is equipped to add more data products in the future.

Previously, when our client received the results from its client surveys, the operations team would have to manually extract some of this information at regular intervals. This semi-automated/semi-manual process required significant resources to maintain. As part of its digital transformation initiatives, the company wanted to automate this process. EPAM’s team of engineers, UX designers and data strategists worked together to help develop a new product that helped our client achieve this goal. Our UX team leveraged telemetry to monitor brain responses from prospective clients to determine which graphics and visuals would be best suited for the new product. We created the look and feel, leveraged new technologies in UI/UX and Delta Lake, and built the new product from end to end.

We helped our client launch an AI-enabled ‘always-on’ tracker that monitors daily attitudes of brands and category dynamics. Not only did the launch allow the company to achieve operational efficiencies through the rule-based, automated system, but it also helped the business move to a subscription-based selling approach. With the enterprise data platform, our client was able to change the way data was being stored and managed so data models could be flexible to capture new data and generate new metrics automatically to support this new subscription-based model.

For our client, data products follow a predictable path: study design, collection, transformation analysis and finally visualization. While we started with the AI-enabled tracker, there are more data products that are in active development that leverage the company’s existing capabilities and architectural blueprints. With the proper foundation now in place, our client can plug in more data products to drastically improve developer productivity and achieve operational efficiencies. With the support of EPAM, the company now has an infrastructure and foundation built for the future.