Partner content: Everything can fail on a data platform migration, and we need to be prepared to avoid high impacts or delays in the plan
Data ecosystems are complex systems of data, technologies, components, applications, code and infrastructure that work together to enable data-driven decision-making. They are increasingly important as organisations seek to harness the power of data to drive innovation, improve efficiency, and gain a competitive advantage.
According to Gartner, data ecosystems are essential for organisations that want to remain competitive in today’s data-driven world. However, building and maintaining a data ecosystem can be challenging. Some of the key challenges include governance, operations costs and scalability.
Move-to-cloud is the answer to all challenges
Cloud adoption is presented by cloud providers with an array of benefits, including cost-efficiency, flexibility, seamless accessibility, and significant reductions in maintenance overheads.
Although the decision to transition to a cloud-based strategy may seem straightforward, the implementation of migration from legacy systems presents a complex challenge by itself and may result in a reality check of experience vs expectations.
The typical quotation that “there is no one-size-fits-all” can also be applied here. Cloud adoption will not be beneficial for all organisations, namely for those not prepared for the paradigm shift of the cloud pricing model or for the possibility that the same processes migrated can have less performance if not adjusted.
It’s crucial to take into account lessons learned from other experiences when deciding to transition to the cloud. I believe that it is in these situations that players such as Celfocus become more relevant, as we help prevent the usual mistakes from being made.
Lessons learned
The key lesson learned is that everything can fail on a data platform migration, and we need to be prepared to avoid high impacts in the plan. And by “be prepared”, I mean leverage on reusable assets that enables the standardisation as much as possible of the migration activities, such as data pipeline generation, testing, data quality, and observability, where we can also include the FinOps.
However, before going into detail on the technical part, it’s important to highlight that the lessons learned start from the strategy and planning moment, and can be split into four main pillars or moments:
> Assessment: Decision on the migration approach based on a detailed technical analysis (Cost, Effort & Benefits) of typical options: Lift & Shift, Lift & Optimise, or Full Refactoring. A typical output of this moment is the target architecture in the cloud and the migration plan.
> Foundations (also known as MVP – Minimum Viable Product): Before jumping to a full migration, it’s important to select the first use case to be migrated. That use case should touch on all the services of the target architecture, to validate all the components’ interoperability and readiness. Thus, we will unlock dependencies for a full migration without having a lot of people waiting at the same time.
> Validation: It’s important to define the acceptance criteria and test strategy from moment zero and evaluate the need to evolve business users on testing and/or create the conditions for parallel run testing, if applicable.
A common challenge in migration projects is the difficulty of accepting the results of a new platform and deciding to decommission the legacy platform, namely in situations where the legacy platform has been used in the same cases for more than 20 years. Another lesson learned is the usage of Automated Testing frameworks to accelerate validations.
> Optimisations: The migration programs will always need an optimisation phase. The best approach is to accept it from the beginning and consider it in the overall planning.
From a technical perspective, Celfocus has learned several lessons from past experiences. Our expertise in Data & Analytics has proven to be effective in achieving success in the move to the cloud. Celfocus is a technology agnostics System Integrator that combines its experience with existing technologies in the market. Regarding data platform migrations to the cloud, we address the challenge in two key topics:
1. Metadata discovery and extraction: The ability to parse existing code from on-prem solutions and extract processes metadata (e.g. sources, transformations, destinations, statistics). The utilisation of a framework becomes more relevant when there is no documentation or low documentation regarding the processes to be migrated.
2. Metadata-driven pipeline generation: A template approach, which is extremely important when we need to create a high number of pipelines that can be fed by metadata to generate physical ETL pipelines on data processing technologies automatically.
For both options, our typical approach is to leverage Celfocus’s data platform frameworks, used to accelerate the project implementation by combining the best practices and previous experiences. For specific situations, we may need to combine them with existing software on the market.
Last but not least, is common to see cost monitoring almost ignored on data platform migration strategies and becoming urgent after a few months or after a bill shock. So, more than just monitoring from the beginning, it’s important to put in place a FinOps strategy, including processes like tagging to allow detailed analysis in the future.
Does the challenge pay off?
In my opinion, yes. In the end, the flexibility obtained by organisations when moving to the cloud is massive. At Celfocus we believe that a data-first approach is the key to successful cloud adoption. This means we should push to have an initial dataset on the cloud, enabling data consumers to interact with the data (e.g. Data Scientists, Data Visualisations, Power Users) and avoiding waiting until the end of the platform migration.
Some of the key advantages are the true data democratisation capabilities provided by a system where it’s easy to:
> Create and destroy a sandbox environment, for example, to validate a use case;
> Decouple storage and computing, allowing multiple applications to access the same data without data duplication;
> Reduce operations effort, allowing teams to focus on what is more important for business.
Finally, it is important to mention that cloud providers are taking the lead on AI and GenAI services, simplifying the deployment and operations of complex algorithms and all the backend infrastructure that would be required for the same goal on-prem.
Embracing the transformative power of AI is essential to unlocking new business opportunities in the future, whether by enhancing customer experiences, optimising operations, or exploring data monetisation.
About the author: João Barata, Data & Analytics Offer Lead at Celfocus
Graduating in Telecommunications and Software Engineering, João Barata is an expert in the Business Intelligence and Analytics areas. For the last decade, he has been delivering projects for different verticals such as Telecommunications, Financial Services, and Utilities, being engaged in development, team leading, and data architecture. Today João is responsible for the Data & Analytics Offer at Celfocus. joao.pedro.barata@celfocus.com