Amazon SQS turns 20: Twenty years of dependable messaging at scale

On July 13, 2006, we launched Amazon Easy Queue Service (Amazon SQS) as one of many first three providers accessible to prospects, alongside Amazon EC2 and Amazon S3. We had discovered firsthand that distributed programs want a dependable method to move messages between parts with out creating tight dependencies. If one service referred to as one other immediately and that service was sluggish or unavailable, failures cascaded by way of the complete system. Message queuing solved this by letting providers talk asynchronously: a producer might drop a message right into a queue and transfer on, whereas a shopper picked it up when prepared. This strategy stored particular person service failures from affecting the remainder of the system.

When Amazon SQS launched publicly in July 2006, it made this sample accessible to each AWS buyer. Twenty years later, that core perform, decoupling producers from customers, stays the explanation prospects use SQS. The size, efficiency, and operational controls round it look very totally different now although.

Jeff Barr coated the primary 15 years of SQS milestones in his fifteenth anniversary submit, from the unique 8 KB message restrict in 2006 by way of FIFO queues, server-side encryption, and Lambda integration. Over the past 5 years, we’ve continued to scale SQS, added stronger safety defaults, and launched new capabilities that handle more and more complicated workload patterns.

Key milestones between 2021 and 2026

Excessive throughput mode for FIFO queues (2021): In Might 2021, we launched common availability of excessive throughput mode for FIFO queues, supporting as much as 3,000 transactions per second (TPS) per API motion, a tenfold improve over the earlier restrict. We continued elevating this ceiling over the next two years: to six,000 TPS in October 2022, to 9,000 TPS in August 2023, and to 18,000 TPS in October 2023, earlier than reaching 70,000 TPS per API motion in choose Areas by November 2023.

Server-side encryption with SSE-SQS (2021): In November 2021, we launched server-side encryption with Amazon SQS-managed encryption keys (SSE-SQS), giving prospects an encryption choice that required no key administration. In October 2022, we made SSE-SQS the default for all newly created queues, so prospects now not wanted to explicitly allow it.

Useless-letter queue redrive enhancements (2021): We progressively expanded how prospects recuperate unconsumed messages from dead-letter queues. In December 2021, we added DLQ redrive to supply queue immediately within the SQS console. In June 2023, we prolonged this functionality to the AWS SDK and CLI by way of new APIs, together with StartMessageMoveTask, CancelMessageMoveTask, and ListMessageMoveTasks. In November 2023, we added redrive assist for FIFO queues.

Attribute-based entry management, ABAC (2022): In November 2022, we launched ABAC, giving prospects the power to configure entry permissions based mostly on queue tags somewhat than sustaining static insurance policies as sources scaled.

JSON protocol assist (2023): In November 2023, we added assist for the JSON protocol within the AWS SDK, decreasing end-to-end message processing latency by as much as 23% for a 5 KB payload and decreasing client-side CPU and reminiscence utilization.

Amazon EventBridge Pipes console integration (2023): We added the power to attach a queue on to EventBridge Pipes from the SQS console, routing messages to a broad vary of AWS service targets with out writing customized integration code.

Prolonged Shopper Library for Python (2024): We introduced the Prolonged Shopper Library, beforehand accessible for Java, to Python builders, permitting messages as much as 2 GB to be despatched by way of SQS by storing the payload in Amazon S3 and passing a reference by way of the queue.

FIFO in-flight message restrict improve (2024): We elevated the in-flight message restrict for FIFO queues from 20,000 to 120,000 messages, so customers can course of considerably extra messages concurrently with out being constrained by the earlier ceiling.

Truthful queues for multi-tenant workloads (2025): We launched truthful queues to mitigate the noisy neighbor downside in multi-tenant customary queues. By together with a message group ID when sending messages, prospects can stop a single tenant from delaying message supply for others, with none adjustments required on the buyer aspect.

1 MiB most message payload measurement (2025): We elevated the utmost message payload from 256 KiB to 1 MiB for each customary and FIFO queues, serving to prospects ship bigger messages with out offloading information to exterior storage. AWS Lambda occasion supply mapping for SQS was up to date in parallel to assist the brand new payload measurement.

The fixed beneath the change

Regardless of 20 years of function additions, the elemental use case for SQS has not shifted. Clients use it to decouple providers, buffer bursts of visitors, and construct programs that keep resilient when particular person parts fail. That very same sample now extends to AI workloads. Clients use SQS queues to buffer requests to massive language fashions, handle inference throughput, and coordinate communication between autonomous AI brokers working as impartial providers. For an instance of this structure in apply, learn Creating asynchronous AI brokers with Amazon Bedrock.

To study extra about Amazon SQS, go to the Amazon SQS product web page, assessment the developer information, or discover latest updates on the AWS Blogs.

— Esra