How Flutter UKI optimizes information pipelines with AWS Managed Workflows for Apache Airflow

This put up is co-written with Monica Cujerean and Ionut Hedesiu from Flutter UKI.

On this put up, we share how Flutter UKI transitioned from a monolithic Amazon Elastic Compute Cloud (Amazon EC2)-based Airflow setup to a scalable and optimized Amazon Managed Workflows for Apache Airflow (Amazon MWAA) structure utilizing options like Kubernetes Pod Operator, steady integration and supply (CI/CD) integration, and efficiency optimization strategies.

About Flutter UKI

As a division of Flutter Leisure, Flutter UKI stands on the forefront of the sports activities betting and gaming trade. Flutter UKI provides a various portfolio of leisure choices, encompassing sports activities wagering, on line casino video games, bingo, and poker experiences. Flutter UKI’s digital presence is strong, working by means of an array of famend on-line manufacturers. These embody the long-lasting Paddy Energy, Sky Betting and Gaming, and Tombola. Whereas Flutter UKI has established a powerful on-line foothold, it maintains a big bodily presence with a community of 576 Paddy Energy betting retailers strategically positioned throughout the UK and Eire.

The Information staff at Flutter UKI is integral to the corporate’s mission of utilizing information to drive enterprise success and innovation. Specializing in information, their groups are devoted to making sure the seamless integration, administration, and accessibility of information throughout a number of sides of the group. By growing strong information pipelines and sustaining excessive information high quality requirements, Flutter UKI empowers stakeholders with dependable insights, optimizes operational efficiencies, and enhances the person expertise. Its dedication to information excellence underpins its efforts to stay on the forefront of the net gaming and leisure trade, delivering worth and strategic benefit to the enterprise.

The journey from self managing Airflow on Amazon EC2 to working Airflow workloads at scale utilizing Amazon MWAA

Flutter UKI’s information orchestration story started in 2017 with a modest Apache Airflow deployment on EC2 situations. As the corporate’s digital footprint expanded, so did their information pipeline necessities, resulting in an more and more advanced monolithic cluster that demanded fixed consideration and useful resource scaling. The operational overhead of managing these EC2 situations grew to become a big problem for his or her engineering groups. In 2022, Flutter UKI reached a crossroads. They wanted to decide on between re-architecting their service on Amazon Elastic Kubernetes Service (Amazon EKS) or embracing Amazon Managed Workflows for Apache Airflow (MWAA).

Flutter UKI was trying to rework their information orchestration service from a resource-intensive, self-managed system to a extra environment friendly, managed service that might permit them to deal with their core enterprise goals somewhat than infrastructure administration. By means of in depth proof-of-concept (POC) testing and shut collaboration with AWS Enterprise Help, Flutter UKI gained confidence within the skill of Amazon MWAA to deal with their subtle workloads at scale. Their alternative of MWAA over a self-managed resolution on Amazon EKS mirrored Flutter UKI’s strategic deal with utilizing managed companies to scale back operational complexity and speed up innovation.

The migration to Amazon MWAA adopted a methodical strategy. There was in depth testing of a number of POCs. Throughout the POCs, the engineering staff discovered MWAA to have an excellent ease of use, which helped them cut back the educational curve leading to quicker. Studying from every POC, they iterated on the ultimate structure by making data-driven choices. Beginning with a small subset of directed acyclic graphs (DAG), the Flutter UKI staff expanded their deployment over time, regularly transferring lots of and finally hundreds of workflows to the managed service. This cautious, phased transition allowed them to validate the efficiency and reliability of MWAA whereas minimizing operational threat.

Excessive-level structure design

Throughout the service re-architecture, the info staff strategically managed over 3,500 dynamically generated DAGs by implementing a classy distribution strategy throughout a number of Amazon MWAA environments to create a workload remoted surroundings. Another excuse for having a number of environments was to make it possible for nobody MWAA surroundings doesn’t get overloaded by a number of DAGs. By putting DAG information throughout various Amazon Easy Storage Service (Amazon S3) places and configuring distinctive DAG_FOLDER paths for every surroundings, the info staff created an clever load balancing mechanism that allocates workflows primarily based on advanced standards together with surroundings sort, job quantity, and environment-specific DAG affinity. A round-robin distribution technique was designed to reduce single surroundings load, guaranteeing scalable infrastructure with zero efficiency degradation. This strategy allowed the staff to optimize workflow orchestration, sustaining excessive efficiency whereas effectively managing an in depth assortment of dynamically generated DAGs throughout a number of MWAA environments. To offer extra compute to particular person duties and to maintain the MWAA environment friendly, Flutter UKI delegated the DAG execution to an exterior compute surroundings utilizing Amazon Elastic Kubernetes Service (Amazon EKS). The ensuing high-level structure is proven within the following determine.

1. Kubernetes Pod Operator (KPO) for duties: Flutter UKI transitioned from utilizing customized operators and plenty of native Airflow operators to completely using the Kubernetes Pod Operator (KPO). This determination simplified their structure by eliminating pointless complexity, decreasing upkeep overhead, and mitigating potential bugs. Moreover, this strategy enabled them to allocate compute sources on a per-task foundation, optimizing total service efficiency. It additionally enabled the usage of completely different container pictures for various duties, thereby avoiding library dependency conflicts.
2. Kubernetes Pod Operator wrapper (KPOw): As an alternative of utilizing KPO immediately, they developed a wrapper (KPOw) round it. This wrapper abstracts the underlying complexity and minimizes the influence of signature adjustments in Airflow, Amazon MWAA, Amazon EKS, or operator variations. By centralizing these adjustments, they solely must replace the wrapper somewhat than hundreds of particular person DAGs. The wrapper additionally simplifies DAGs by hiding repetitive parameters, akin to node affinity, pod sources, and EKS cluster configurations. Moreover, it enforces company-specific naming conventions and permits for parameter validation at job execution time somewhat than throughout DagBag refresh. In addition they launched profiles and picture information, the place profile information include obligatory KPO parameters, and the corresponding picture information hyperlink to the repository for the duty’s container picture. This setup ensures consistency throughout duties utilizing the identical profile and facilitates simultaneous updates throughout duties.
3. Month-to-month picture updates in Kubernetes: Imposing a coverage of month-to-month picture updates made certain that their code remained present, stopping safety vulnerabilities and avoiding in depth code adjustments on account of deprecated libraries.
4. Steady Airflow updates: Flutter UKI maintains a cutting-edge infrastructure by implementing new Airflow variations shortly after launch, whereas following a fastidiously orchestrated deployment technique. Their strategy makes use of customary Amazon MWAA configurations and employs a scientific testing protocol. New variations are first deployed to growth and take a look at environments for thorough validation earlier than reaching manufacturing techniques. This methodical development considerably reduces the chance of disruptions to business-critical workflows.

To realize operational excellence, Flutter UKI has carried out a complete monitoring framework centered on Amazon CloudWatch metrics. Their monitoring resolution contains strategically configured alarms that present early warning alerts for potential points. This proactive monitoring strategy allows their groups to rapidly determine and examine anomalies in manufacturing workload executions, guaranteeing excessive availability and efficiency of their information pipelines. The mix of cautious model administration and strong monitoring exemplifies Flutter UKI’s dedication to operational excellence of their cloud infrastructure.

1. CI/CD integration: By managing their code in GitLab, with obligatory code evaluations and utilizing Argo Occasions and Argo Workflows for picture updates in AWS ECR, they streamlined their growth processes.
2. Efficiency Optimization: A good portion of the DAGs are dynamically generated primarily based on database metadata. This era course of runs outdoors Amazon MWAA, with its personal CI/CD pipeline, and the ensuing DAG information are saved within the S3 DAG. Inserting code outdoors of duties was prevented, together with parameter analysis. Parameters and secrets and techniques are saved in AWS Secrets and techniques Supervisor and retrieved at job runtime. Engineers purpose to reduce or eradicate inter-service dependencies inside MWAA.

DAGs are scheduled to distribute execution occasions as evenly as attainable. Process code and customary modules are hosted on Amazon S3 and retrieved at runtime. For bigger codebases, Amazon Elastic File System (Amazon EFS) volumes are mounted to job pods are used.

Outcomes

Right this moment, Flutter UKI’s infrastructure includes 4 Amazon MWAA clusters, every executing duties on devoted Amazon EKS node teams. They handle roughly 5,500 DAGs encompassing over 30,000 duties, dealing with greater than 60,000 DAG runs each day with a concurrency exceeding 450 duties operating concurrently throughout clusters. They anticipate a ten% month-to-month improve on this workload within the brief to medium time period. Throughout main occasions like Cheltenham and Grand Nationwide, the place information load will increase by 30%, their MWAA service has demonstrated stability and scalability, attaining a 100% success charge for crucial processes in 2025, a big enchancment over earlier years.

Conclusion

Flutter UKI’s journey with AWS Managed Workflows for Apache Airflow (Amazon MWAA) has resulted in a steady, scalable, and resilient manufacturing surroundings. The cautious re-architecting of Flutter UKI’s service, mixed with strategic choices round job execution and infrastructure administration, has not solely simplified their operations, but additionally enhanced efficiency and reliability. Safety and compliance advantages had been additionally seen, as a result of MWAA supplies managed safety updates, built-in encryption, and integration with AWS safety companies. Maybe most significantly, the shift to MWAA has allowed Flutter UKI’s engineering groups to redirect their efforts from infrastructure upkeep to business-critical duties, specializing in DAG growth and bettering information pipeline effectivity, finally accelerating innovation of their core enterprise operations.

In the event you’re trying to cut back operational overhead and migrate to a totally managed Airflow resolution on AWS, think about using Amazon MWAA. Get in contact together with your Technical Account Supervisor or your Options Architect to debate an answer particular to your use-case. You too can attain out to AWS Help by making a case when you’re dealing with an points establishing the service.

Able to see what Amazon MWAA is like? Go to the AWS Administration Console for Amazon MWAA. For extra info, see What Is Amazon Managed Workflows for Apache Airflow. Moreover, Utilizing Amazon MWAA with Amazon EKS reveals you how one can combine Amazon MWAA with Amazon EKS.

In regards to the authors

Monica Cujerean is a Principal Information Engineer at Flutter UKI, specializing in service associated initiatives that cowl efficiency optimization, value effectiveness, and new function adoption on most AWS service in our stack: Amazon MWAA, Amazon Redshift, Amazon Aurora, and Amazon SageMaker.

Ionut Hedesiu is a Senior Information Architect at Flutter UKI, chargeable for designing strategic options to cowl advanced and assorted enterprise wants. His major experience is on Amazon MWAA, Kubernetes, Amazon Sagemaker, and ETL options.

Nidhi Agrawal is a Technical Account Supervisor at AWS and works with giant enterprise prospects to supply the technical steering, greatest practices, and strategic assist to prospects, serving to them optimize their environments within the AWS Cloud.

John Kellett is a Senior Buyer Options Supervisor with 25 years of expertise throughout personal and public sectors. John helps drive end-to-end buyer engagement by means of program administration excellence. By understanding and representing prospects’ strategic visions, John aligns to develop the individuals, organizational readiness, and know-how competencies to fulfill the specified outcomes.

Sidhanth Muralidhar is a Principal Technical Account Supervisor at AWS. He works with giant enterprise prospects who run their workloads on AWS. He’s captivated with working with prospects and serving to them architect workloads for value, reliability, efficiency, and operational excellence at scale of their cloud journey. He has a eager curiosity in information analytics as nicely.