Introducing workload simulation workbench for Amazon MSK Categorical dealer

April 8, 2026

2

Validating Kafka configurations earlier than manufacturing deployment will be difficult. On this put up, we introduce the workload simulation workbench for Amazon Managed Streaming for Apache Kafka (Amazon MSK) Categorical Dealer. The simulation workbench is a device that you should use to securely validate your streaming configurations by means of practical testing situations.

Answer overview

Various message sizes, partition methods, throughput necessities, and scaling patterns make it difficult so that you can predict how your Apache Kafka configurations will carry out in manufacturing. The normal approaches to check these variables create vital boundaries: ad-hoc testing lacks consistency, handbook arrange of momentary clusters is time-consuming and error-prone, production-like environments require devoted infrastructure groups, and workforce coaching typically occurs in isolation with out practical situations. You want a structured strategy to check and validate these configurations safely earlier than deployment. The workload simulation workbench for MSK Categorical Dealer addresses these challenges by offering a configurable, infrastructure as code (IaC) resolution utilizing AWS Cloud Growth Package (AWS CDK) deployments for practical Apache Kafka testing. The workbench helps configurable workload situations, and real-time efficiency insights.

Categorical brokers for MSK Provisioned make managing Apache Kafka extra streamlined, less expensive to run at scale, and extra elastic with the low latency that you simply count on. Every dealer node can present as much as 3x extra throughput per dealer, scale as much as 20x sooner, and get well 90% faster in comparison with normal Apache Kafka brokers. The workload simulation workbench for Amazon MSK Categorical dealer facilitates systematic experimentation with constant, repeatable outcomes. You should use the workbench for a number of use circumstances like manufacturing capability planning, progressive coaching to arrange builders for Apache Kafka operations with rising complexity, and structure validation to show streaming designs and examine totally different approaches earlier than making manufacturing commitments.

Structure overview

The workbench creates an remoted Apache Kafka testing atmosphere in your AWS account. It deploys a personal subnet the place shopper and producer functions run as containers, connects to a personal MSK Categorical dealer and screens for efficiency metrics and visibility. This structure mirrors the manufacturing deployment sample for experimentation. The next picture describes this structure utilizing AWS providers.

MSK Workload SImulator WorkBench Architecture Diagram

This structure is deployed utilizing the next AWS providers:

Amazon Elastic Container Service (Amazon ECS) generate configurable workloads with Java-based producers and customers, simulating varied real-world situations by means of totally different message sizes and throughput patterns.

Amazon MSK Categorical Cluster runs Apache Kafka 3.9.0 on Graviton-based cases with hands-free storage administration and enhanced efficiency traits.

Dynamic Amazon CloudWatch Dashboards robotically adapt to your configuration, displaying real-time throughput, latency, and useful resource utilization throughout totally different check situations.

Safe Amazon Digital Non-public Cloud (Amazon VPC) Infrastructure offers personal subnets throughout three Availability Zones with VPC endpoints for safe service communication.

Configuration-driven testing

The workbench offers totally different configuration choices in your Apache Kafka testing atmosphere, so you’ll be able to customise occasion varieties, dealer rely, matter distribution, message traits, and ingress fee. You may regulate the variety of subjects, partitions per matter, sender and receiver service cases, and message sizes to match your testing wants. These versatile configurations assist two distinct testing approaches to validate totally different features of your Kafka deployment:

Method 1: Workload validation (single deployment)

Take a look at totally different workload patterns towards the identical MSK Categorical cluster configuration. That is helpful for evaluating partition methods, message sizes, and cargo patterns.

// Mounted MSK Categorical Cluster Configuration
export const mskBrokerConfig: MskBrokerConfig = {
numberOfBrokers: 1, // 1 dealer per AZ = 3 whole brokers
instanceType: 'categorical.m7g.giant', // MSK Categorical occasion kind
};

// A number of Concurrent Workload Exams
export const deploymentConfig: DeploymentConfig = { providers: [
{ topics: 2, partitionsPerTopic: 6, instances: 3, messageSizeBytes: 1024 }, // High-throughput scenario
{ topics: 1, partitionsPerTopic: 3, instances: 1, messageSizeBytes: 512 }, // Latency-optimized scenario
{ topics: 3, partitionsPerTopic: 4, instances: 2, messageSizeBytes: 4096 }, // Multi-topic scenario
]};

Method 2: Infrastructure rightsizing (redeploy and examine)

Take a look at totally different MSK Categorical cluster configurations by redeploying the workbench with totally different dealer settings whereas conserving the identical workload. That is really helpful for rightsizing experiments and understanding the influence of vertical in comparison with horizontal scaling.

// Baseline: Deploy and check
export const mskBrokerConfig: MskBrokerConfig = { numberOfBrokers: 1, instanceType: 'categorical.m7g.giant',};

// Vertical scaling: Redeploy with bigger cases
export const mskBrokerConfig: MskBrokerConfig = { numberOfBrokers: 1,
instanceType: 'categorical.m7g.xlarge', // Bigger cases
};

// Horizontal scaling: Redeploy with extra brokers
export const mskBrokerConfig: MskBrokerConfig = {
numberOfBrokers: 2, // Extra brokers
instanceType: 'categorical.m7g.giant',};

Every redeployment makes use of the identical workload configuration, so you’ll be able to isolate the influence of infrastructure adjustments on efficiency.

Workload testing situations (single deployment)

These situations check totally different workload patterns towards the identical MSK Categorical cluster:

Partition technique influence testing

Situation: You might be debating the utilization of fewer subjects with many partitions in comparison with many subjects with fewer partitions in your microservices structure. You need to perceive how partition rely impacts throughput and shopper group coordination earlier than making this architectural resolution.

const deploymentConfig = { providers: [
{ topics: 1, partitionsPerTopic: 1, instances: 2, messageSizeBytes: 1024 }, // Baseline: minimal partitions
{ topics: 1, partitionsPerTopic: 10, instances: 2, messageSizeBytes: 1024 }, // Medium partitions
{ topics: 1, partitionsPerTopic: 20, instances: 2, messageSizeBytes: 1024 }, // High partitions
]};

Message dimension efficiency evaluation

Situation: Your utility handles several types of occasions – small IoT sensor readings (256 bytes), medium person exercise occasions (1 KB), and enormous doc processing occasions (8KB). You should perceive how message dimension impacts your general system efficiency and for those who ought to separate these into totally different subjects or deal with them collectively.

const deploymentConfig = { providers: [
{ topics: 2, partitionsPerTopic: 6, instances: 3, messageSizeBytes: 256 }, // IoT sensor data
{ topics: 2, partitionsPerTopic: 6, instances: 3, messageSizeBytes: 1024 }, // User events
{ topics: 2, partitionsPerTopic: 6, instances: 3, messageSizeBytes: 8192 }, // Document events
]};

Load testing and scaling validation

Situation: You count on visitors to range considerably all through the day, with peak masses requiring 10× extra processing capability than off-peak hours. You need to validate how your Apache Kafka subjects and partitions deal with totally different load ranges and perceive the efficiency traits earlier than manufacturing deployment.

const deploymentConfig = { providers: [
{ topics: 2, partitionsPerTopic: 6, instances: 1, messageSizeBytes: 1024 }, // Off-peak load simulation
{ topics: 2, partitionsPerTopic: 6, instances: 5, messageSizeBytes: 1024 }, // Medium load simulation
{ topics: 2, partitionsPerTopic: 6, instances: 10, messageSizeBytes: 1024 }, // Peak load simulation
]};

Infrastructure rightsizing experiments (redeploy and examine)

These situations assist you perceive the influence of various MSK Categorical cluster configurations by redeploying the workbench with totally different dealer settings:

MSK dealer rightsizing evaluation

Situation: You deploy a cluster with primary configuration and put load on it to ascertain baseline efficiency. Then you definitely need to experiment with totally different dealer configurations to see the impact of vertical scaling (bigger cases) and horizontal scaling (extra brokers) to seek out the best cost-performance steadiness in your manufacturing deployment.

Step 1: Deploy with baseline configuration

// Preliminary deployment: Fundamental configuration
export const mskBrokerConfig: MskBrokerConfig = {
numberOfBrokers: 1, // 3 whole brokers (1 per AZ)
instanceType: 'categorical.m7g.giant',};export const deploymentConfig: DeploymentConfig = { providers: [ { topics: 2, partitionsPerTopic: 6, instances: 3, messageSizeBytes: 1024 }, ]};

Step 2: Redeploy with vertical scaling

// Redeploy: Take a look at vertical scaling influence
export const mskBrokerConfig: MskBrokerConfig = {
numberOfBrokers: 1, // Identical dealer rely
instanceType: 'categorical.m7g.xlarge', // Bigger cases
};

// Preserve similar workload configuration to match outcomes

Step 3: Redeploy with horizontal scaling

// Redeploy: Take a look at horizontal scaling influence
export const mskBrokerConfig: MskBrokerConfig = {
numberOfBrokers: 2, // 6 whole brokers (2 per AZ)
instanceType: 'categorical.m7g.giant', // Again to authentic dimension
};

// Preserve similar workload configuration to match outcomes

This rightsizing method helps you perceive how dealer configuration adjustments have an effect on the identical workload, so you’ll be able to enhance each efficiency and value in your particular necessities.

Efficiency insights

The workbench offers detailed insights into your Apache Kafka configurations by means of monitoring and analytics, making a CloudWatch dashboard that adapts to your configuration. The dashboard begins with a configuration abstract displaying your MSK Categorical cluster particulars and workbench service configurations, serving to you to know what you’re testing. The next picture exhibits the dashboard configuration abstract:

The second part of dashboard exhibits real-time MSK Categorical cluster metrics together with:

Dealer efficiency: CPU utilization and reminiscence utilization throughout brokers in your cluster
Community exercise: Monitor bytes in/out and packet counts per dealer to know community utilization patterns
Connection monitoring: Shows energetic connections and connection patterns to assist determine potential bottlenecks
Useful resource utilization: Dealer-level useful resource monitoring offers insights into general cluster well being

The next picture exhibits the MSK cluster monitoring dashboard:

The third part of the dashboard exhibits the Clever Rebalancing and Cluster Capability insights displaying:

Clever rebalancing: in progress: Reveals whether or not a rebalancing operation is at present in progress or has occurred prior to now. A price of 1 signifies that rebalancing is actively working, whereas 0 signifies that the cluster is in a gradual state.
Cluster under-provisioned: Signifies whether or not the cluster has inadequate dealer capability to carry out partition rebalancing. A price of 1 signifies that the cluster is under-provisioned and Clever Rebalancing can’t redistribute partitions till extra brokers are added or the occasion kind is upgraded.
International partition rely: Shows the whole variety of distinctive partitions throughout all subjects within the cluster, excluding replicas. Use this to trace partition progress over time and validate your deployment configuration.
Chief rely per dealer: Reveals the variety of chief partitions assigned to every dealer. An uneven distribution signifies partition management skew, which might result in hotspots the place sure brokers deal with disproportionate learn/write visitors.
Partition rely per dealer: Reveals the whole variety of partition replicas hosted on every dealer. This metric contains each chief and follower replicas and is essential to figuring out reproduction distribution imbalances throughout the cluster.

The next picture exhibits the Clever Rebalancing and Cluster Capability part of the dashboard:

The fourth part of the dashboard exhibits the application-level insights displaying:

System throughput: Shows the whole variety of messages per second throughout providers, providing you with an entire view of system efficiency
Service comparisons: Performs side-by-side efficiency evaluation of various configurations to know which approaches match
Particular person service efficiency: Every configured service has devoted throughput monitoring widgets for detailed evaluation
Latency evaluation: The top-to-end message supply occasions and latency comparisons throughout totally different service configurations
Message dimension influence: Efficiency evaluation throughout totally different payload sizes helps you perceive how message dimension impacts general system conduct

The next picture exhibits the applying efficiency metrics part of the dashboard:

Getting began

This part walks you thru establishing and deploying the workbench in your AWS atmosphere. You’ll configure the mandatory conditions, deploy the infrastructure utilizing AWS CDK, and customise your first check.

Conditions

You may deploy the answer from the GitHub Repo. You may clone it and run it in your AWS atmosphere. To deploy the artifacts, you’ll require:

AWS account with administrative credentials configured for creating AWS assets.
AWS Command Line Interface (AWS CLI) should be configured with acceptable permissions for AWS useful resource administration.
AWS Cloud Growth Package (AWS CDK) needs to be put in globally utilizing npm set up -g aws-cdk for infrastructure deployment.
Node.js model 20.9 or larger is required, with model 22+ really helpful.
Docker engine should be put in and working domestically because the CDK builds container photographs throughout deployment. Docker daemon needs to be working and accessible to CDK for constructing the workbench utility containers.

Deployment

# Clone the workbench repository
git clone https://github.com/aws-samples/sample-simulation-workbench-for-msk-express-brokers.git

# Set up dependencies and construct
npm set up 
npm run construct

# Bootstrap CDK (first time solely per account/area)
cd cdk 
npx cdk bootstrap

# Synthesize CloudFormation template (non-obligatory verification step)
npx cdk synth

# Deploy to AWS (creates infrastructure and builds containers)
npx cdk deploy

After deployment is accomplished, you’ll obtain a CloudWatch dashboard URL to watch the workbench efficiency in real-time.You too can deploy a number of remoted cases of the workbench in the identical AWS account for various groups, environments, or testing situations. Every occasion operates independently with its personal MSK cluster, ECS providers, and CloudWatch dashboards.To deploy extra cases, modify the Surroundings Configuration in cdk/lib/config.ts:

// Occasion 1: Growth workforce
export const AppPrefix = 'mske';export const EnvPrefix = 'dev';

// Occasion 2: Staging atmosphere (separate deployment)
export const AppPrefix = 'mske';export const EnvPrefix = 'staging';

// Occasion 3: Crew-specific testing (separate deployment)
export const AppPrefix = 'team-alpha';export const EnvPrefix = 'check';

Every mixture of AppPrefix and EnvPrefix creates fully remoted AWS assets in order that a number of groups or environments can use the workbench concurrently with out conflicts.

Customizing your first check

You may edit the configuration file situated at folder “cdk/lib/config-types.ts” to outline your testing situations and run the deployment. It’s preconfigured with the next configuration:

export const deploymentConfig: DeploymentConfig = { providers: [
// Start with a simple baseline test
{ topics: 1, partitionsPerTopic: 3, instances: 1, messageSizeBytes: 1024 },

// Add a comparison scenario
{ topics: 1, partitionsPerTopic: 6, instances: 1, messageSizeBytes: 1024 }, ]};

Greatest practices

Following a structured method to benchmarking ensures that your outcomes are dependable and actionable. These greatest practices will assist you isolate efficiency variables and construct a transparent understanding of how every configuration change impacts your system’s conduct. Start with single-service configurations to ascertain baseline efficiency:

const deploymentConfig = { providers: [ { topics: 1, partitionsPerTopic: 3, instances: 1, messageSizeBytes: 1024 } ]};

After you perceive the baseline, add comparability situations.

Change one variable at a time

For clear insights, modify just one parameter between providers:

const deploymentConfig = { providers: [
{ topics: 1, partitionsPerTopic: 3, instances: 1, messageSizeBytes: 1024 }, // Baseline
{ topics: 1, partitionsPerTopic: 6, instances: 1, messageSizeBytes: 1024 }, // More partitions
{ topics: 1, partitionsPerTopic: 12, instances: 1, messageSizeBytes: 1024 }, // Even more partitions
]};

This method helps you perceive the influence of particular configuration adjustments.

Essential concerns and limitations

Earlier than counting on workbench outcomes for manufacturing selections, you will need to perceive the device’s meant scope and bounds. The next concerns will assist you set acceptable expectations and make the best use of the workbench in your planning course of.

Efficiency testing disclaimer

The workbench is designed as an academic and sizing estimation device to assist groups put together for MSK Categorical manufacturing deployments. Whereas it offers useful insights into efficiency traits:

Outcomes can range primarily based in your particular use circumstances, community situations, and configurations
Use workbench outcomes as steerage for preliminary sizing and planning
Conduct complete efficiency validation together with your precise workloads in production-like environments earlier than ultimate deployment

Beneficial utilization method

Manufacturing readiness coaching – Use the workbench to arrange groups for MSK Categorical capabilities and operations.

Structure validation – Take a look at streaming architectures and efficiency expectations utilizing MSK Categorical enhanced efficiency traits.

Capability planning – Use MSK Categorical streamlined sizing method (throughput-based somewhat than storage-based) for preliminary estimates.

Crew preparation – Construct confidence and experience with manufacturing Apache Kafka implementations utilizing MSK Categorical.

Conclusion

On this put up, we confirmed how the workload simulation workbench for Amazon MSK Categorical Dealer helps studying and preparation for manufacturing deployments by means of configurable, hands-on testing and experiments. You should use the workbench to validate configurations, construct experience, and enhance efficiency earlier than manufacturing deployment. In the event you’re making ready in your first Apache Kafka deployment, coaching a workforce, or enhancing current architectures, the workbench offers sensible expertise and insights wanted for fulfillment. Check with Amazon MSK documentation – Full MSK Categorical documentation, greatest practices, and sizing steerage for extra data.