Uma arquitetura de código aberto para a aplicação do Strangler Pattern com Change Data Capture para setores com alto volume de dados

Abstract

This paper proposes an open-source architecture for implementing the Strangler Pattern in contexts managing large volumes of data, using technologies such as Apache Spark, Debezium, and Apache Kafka. The Strangler Pattern is a legacy system modernization strategy that facilitates the gradual transition to new functionalities and services, minimizing the disruption of essential services. It is particularly beneficial in scenarios dealing with large data volumes, where continuity and evolution of services need to be carefully balanced. In this architecture, Apache Spark and Debezium are employed to implement the Changed Data Capture (CDC), a technique that identifies and captures changes in a database, including INSERT, UPDATE, and DELETE operations. This allows for processing and transfer of only modified data to the Data Warehouse, Data Lake, or other destinations, optimizing processing time and resource use. The architecture also utilizes Apache Kafka to manage real-time data flows, a critical feature for business intelligence in high data volume environments. Other tools such as Kafka Connect and Alluxio play central roles in the architecture and will be discussed later. Initial results suggest that the proposed architecture can significantly decrease computational costs, improve data processing efficiency, and minimize the disruption of existing services. The paper thoroughly presents the architecture implementation, addressing challenges, solutions, and a case study in the supply chain market where the architecture was initially applied.