How Alight Options achieved 55% price financial savings with Amazon OpenSearch Service

This can be a visitor submit by Mark Larson, Andrew Kummerow, and Tim Razik at Alight Options, in partnership with AWS.

Alight Options is a number one cloud-based human capital expertise and providers supplier targeted on built-in advantages administration, healthcare navigation, and worker expertise options. The corporate serves a whole bunch of enterprise clients globally, with providers that help tens of millions of individuals worldwide.

Alight’s expertise stack generates over 1 billion log information per day throughout their containerized microservices structure, with peaks reaching 100,000 information per second throughout Annual Enrollment intervals. Beforehand, Alight relied on a self-managed Elastic Stack (Elasticsearch, Logstash, Kibana) deployment that had been in manufacturing since 2018. As their logging volumes grew and Elasticsearch 7.x approached finish of help, the operational burden of sustaining this infrastructure consumed their whole operational finances, leaving no capability for innovation.

On this submit, we share how Alight Options migrated from self-managed Elasticsearch to Amazon OpenSearch Service. The migration achieved a 55% price discount, alleviated roughly 2,000 hours per yr of operational overhead, and gave Alight entry to superior observability options they might not prioritize earlier than.

Challenges with self-managed Elasticsearch

Alight’s self-managed Elastic Stack infrastructure introduced compounding technical and operational challenges. Their manufacturing surroundings consisted of 15 Elasticsearch nodes with 168 TB of EBS storage, dealing with log ingestion from their flagship Alight Worklife system and supporting purposes. The infrastructure required an Elastic Platinum subscription, although the group’s operational bandwidth was totally consumed by upkeep, leaving restricted capability to undertake superior options included within the license.

The operational ache factors included:

• Safety vulnerability patching required working over Christmas holidays to handle crucial fixes, with no flexibility on timing.
• Elastic upgrades have been time-consuming and required depth of data to handle at scale.
• Logstash utilizing TCP-socket transport was unreliable, experiencing log loss at excessive volumes.
• Backpressure from Logstash prompted two P1 incidents over two years, the place the logging subsystem immediately impacted microservice duties.
• Elasticsearch 7.x approaching finish of help created urgency to behave earlier than the following Annual Enrollment interval (September by way of January).

Alight was spending greater than $100,000 per 30 days on self-managed infrastructure and Elastic licensing throughout all environments. All operational finances was consumed by cluster upkeep, leaving zero capability for innovation.

Evaluating options

Alight evaluated a number of options earlier than deciding on OpenSearch Service:

• New Relic and Dynatrace have been evaluated for log aggregation however proved prohibitively costly at Alight’s quantity.
• Amazon CloudWatch was evaluated however didn’t meet necessities for complicated log analysis at their quantity and visualization complexity.

Amazon OpenSearch Service is a managed service that makes it easy to deploy, function, and scale OpenSearch clusters within the AWS Cloud. You need to use it to be used circumstances resembling log analytics and real-time software monitoring. It provisions cluster assets, routinely detects and replaces failed nodes, and scales with a single API name or a couple of clicks, lowering the operational overhead related to self-managed infrastructure. It gained the analysis primarily based on 5 components:

1. Price: considerably cheaper than self-managed Elastic Stack and competing options.
2. Minimal change administration: as a fork of Elasticsearch 7.10, engineers have been already acquainted with the question syntax and dashboards.
3. Compliance: utilizing a local AWS service prevented a whole bunch of hours of vendor compliance, audit, and regulatory work. The group spent a couple of hours getting approval in comparison with doubtlessly weeks for an exterior vendor.
4. Cloud-native technique: aligned with Alight’s overarching technique to make use of cloud-native providers.
5. Safety and information privateness: holding all the things inside their AWS touchdown zone alleviated information egress considerations.

Resolution overview

Alight partnered with AWS to design a cloud-native log aggregation structure that changed self-managed Elasticsearch and Logstash with Amazon OpenSearch Service and Amazon OpenSearch Ingestion (OSIS), assuaging the operational burden, together with the Logstash backpressure that had prompted two P1 incidents.

The structure makes use of a cross-account mannequin with two major account varieties:

The next diagram illustrates the answer structure.

Alight OpenSearch Service structure exhibiting cross-account log ingestion from Amazon ECS and Amazon EC2 workloads by way of OpenSearch Ingestion to Amazon OpenSearch Service

Ingestion paths

The answer helps a number of ingestion paths relying on the applying internet hosting mannequin:

• ECS purposes: FireLens/Fluent Bit sidecar containers seize stdout/stderr by way of the awsfirelens log driver, then ship logs over HTTPS on to OSIS within the shared providers account. ECS job roles assume a cross-account OSIS Ingest Position for authentication.
• EC2 purposes: Open-source Fluent Bit (RPM-based, non-containerized) makes use of tail enter to learn log recordsdata, then ships to OSIS by way of an EC2 IAM Position with cross-account belief.
• S3-based ingestion (deliberate): Some purposes write to Amazon Easy Storage Service (Amazon S3) with Amazon Easy Queue Service (Amazon SQS) notifications triggering OSIS pipelines.

Spring Boot microservices use a customized logging framework constructed on Logback (not Log4j) that codecs logs as JSON and flushes to console, which FireLens picks up.

Safety mannequin

Visitors flows over HTTPS. The safety mannequin makes use of position separation with least privilege:

• OSIS Ingest Position: write-only entry to OSIS pipelines, assumed by software account roles by way of cross-account belief.
• OSIS Sink Position: utilized by OSIS to write down into the OpenSearch area, with full index entry scoped to the ingestion pipeline.
• Safety teams: limit OSIS visitors to recognized CIDRs and VPCs.

Every software has its personal indices, and entry is ruled by application-specific roles.

Persistent buffering

Amazon Elastic File System (Amazon EFS) supplies persistent filesystem buffering for the Fluent Bit sidecar, serving to stop log loss throughout transient failures or backpressure occasions. This immediately addresses the P1 incidents Alight skilled with Logstash. For the following Annual Enrollment interval, Alight plans to additionally allow persistent buffering on the OSIS layer to deal with burst ingestion with out log loss.

Consumer entry

Finish-user entry to OpenSearch Dashboards is managed by way of AWS IAM Id Heart with System for Cross-domain Id Administration (SCIM) synchronization from Alight’s enterprise Id Supplier. Customers navigate to the Functions tab in Id Heart to entry OpenSearch Dashboards over SAML/HTTPS.

At Alight, IAM Id Heart and SCIM are configured within the payer account. They use the identical synchronization and entitlement request and approval course of that governs Alight’s person and entitlement provisioning into AWS. With this setup, the group makes use of the identical single sign-on (SSO) and entitlement workflow for OpenSearch Dashboards entry as for the AWS Administration Console, along with fine-grained entry management (FGAC) outlined throughout the OpenSearch domains.

OpenSearch area configuration

For his or her manufacturing workload, Alight deployed:

Further environments embody a secondary manufacturing cluster (12 sizzling nodes, 3 UltraWarm, 3 devoted chief nodes), plus consumer check and engineering clusters with 3 sizzling nodes every.

Migration course of

The migration was accomplished over seven months (February by way of August 2025), with 5 purposes migrated together with the flagship Alight Worklife software.

Infrastructure as code

The group constructed new Terraform modules to handle deployment of OSIS pipelines, OpenSearch domains, and FireLens sidecar additions to ECS purposes. Onboarding new purposes is now templatized, leading to vital time financial savings in comparison with including new indices in Elasticsearch. Onboarding a brand new software now takes between 4-8 hours, whereas earlier than we’d spend 80-120 hours per software.

Migration timeline

Alight first enabled Amazon OpenSearch Service in manufacturing for 2 smaller purposes, to ensure operational processes have been up and operating earlier than migrating the best quantity log producers. For every software, logging to OpenSearch was enabled whereas persevering with to write down logs to the prevailing logging infrastructure. This parallel run allowed fine-tuning of OSIS pipeline configuration, OpenSearch cluster measurement and configuration earlier than doing a full cutover. This strategy additionally validated that logs have been being ingested correctly into OpenSearch. It confirmed that the efficiency of OpenSearch Dashboards and queries was pretty much as good as or higher than the prevailing self-managed Elasticsearch cluster.

For historic information, Alight migrated the newest 30 days of reside information from Elasticsearch into OpenSearch simply previous to cutover. In addition they retained a full archive of older log information in an Amazon S3 bucket, in order that information older than 30 days may very well be loaded into OpenSearch on request if a person wants it.

AWS partnership

Alight engaged the AWS group in the course of the analysis section. By way of AWS Enterprise Help, their Technical Account Supervisor (TAM) served because the devoted level of contact all through the journey. The TAM coordinated classes with OpenSearch Service material consultants to handle particular service capabilities, assist with design, troubleshoot points, and supply efficiency steering.

Outcomes

The migration to Amazon OpenSearch Service delivered outcomes throughout price, operations, and functionality dimensions.

“Alight’s mission crucial purposes are constructed on a whole bunch of interdependent microservices, so efficient software logging is crucial for analyzing system behaviors, efficiency tuning, and troubleshooting. Amazon OpenSearch Service supplies us with nice log analytics, very affordably at scale, and integrates seamlessly with our IAM technique for granular entry management and authorization. The flexibility to reconfigure, resize, and improve OpenSearch domains with a couple of clicks and 0 downtime is a sport changer for us.”

— Mark Larson, Enterprise Architect

Price and licensing

Not all Elasticsearch clusters are decommissioned but. As soon as decommissioning is full, financial savings will attain roughly 65%. Moreover, extra purposes have been added to OpenSearch than have been initially on Elasticsearch, making the per-application price much more favorable. Past compute and licensing, the migration additionally lowered information switch prices beforehand incurred throughout the self-managed cross-account structure, including additional to the general financial savings.

Operational enhancements

Efficiency and scale

Classes realized and finest practices

By way of their migration journey, Alight gained the next insights:

• Use your account group relationship to advocate: When Fluent Bit had a blocking situation, the AWS account group relationship helped push for the repair and offered workaround steering.
• Separate considerations for information sturdiness: Don’t put 100% supply ensures on logging infrastructure. Use a separate occasion stream (resembling Amazon SQS) for crucial information that can’t tolerate loss.
• Templatize all the things: Terraform modules for OSIS, OpenSearch domains, and FireLens sidecars cut back the time to onboard new purposes.
• Safety structure issues: Separating ingest roles from sync roles (least privilege) and utilizing cross-account belief supplies sturdy safety with out complexity.
• Plan round business-critical intervals: Pausing the manufacturing rollout throughout Annual Enrollment was the correct name. The danger of introducing modifications throughout peak was not well worth the schedule stress.

What’s subsequent

Alight has a number of initiatives deliberate to broaden their OpenSearch Service utilization:

• Anomaly detection: high precedence, a characteristic they paid for with Elastic Platinum however by no means had capability to implement.
• Amazon OpenSearch Serverless: evaluating for brand new log sources, significantly all for zero-OCU baseline for price optimization.
• OSIS persistent buffer: deliberate for subsequent Annual Enrollment to deal with burst ingestion with out log loss.
• Amazon Bedrock AgentCore logging: new synthetic intelligence (AI) workloads will ship logs to OpenSearch.
• AI-assisted log analytics: adopting the agentic AI capabilities now constructed into Amazon OpenSearch Service. These embody the Investigation Agent for autonomous, hypothesis-driven root trigger evaluation, which helps website reliability engineering (SRE) and engineering groups achieve deeper insights from software logs.
• Vector database: already utilizing OpenSearch as a vector retailer for a conversational AI assistant (separate group).
• Migration progress: All workloads beforehand logging to Elasticsearch have been migrated to OpenSearch, plus an extra eight purposes.
• Enterprise Logging Service: All new purposes will now log to Amazon OpenSearch Service by default utilizing the templatized strategy.
• Decommission: All current Elasticsearch situations can be decommissioned by July 2026.

Conclusion

Alight’s migration from self-managed Elasticsearch to Amazon OpenSearch Service demonstrates how enterprises can alleviate operational burden whereas reaching vital price financial savings. By utilizing Amazon OpenSearch Ingestion and FireLens, Alight constructed a scalable log aggregation system that handles 1 billion information per day with zero P1 incidents since deployment.

The 55% price discount and roughly 2,000 hours per yr of recovered engineering time have freed Alight to pursue superior observability capabilities like anomaly detection and AI-powered log analytics, options they paid for however might by no means use below the operational weight of self-managed infrastructure.

To be taught extra, see the Amazon OpenSearch Service documentation. To get began with ingestion pipelines, see Amazon OpenSearch Ingestion. For migration steering, see Migrating to Amazon OpenSearch Service.

Concerning the authors