Accelerating CI with AWS CodeBuild: Parallel check execution now accessible

I’m excited to announce that AWS CodeBuild now helps parallel check execution, so you’ll be able to run your check suites concurrently and cut back construct instances considerably.

With the demo undertaking I wrote for this put up, the entire check time went down from 35 minutes to six minutes, together with the time to provision the environments. These two screenshots from the AWS Administration Console present the distinction.

Sequential execution of the check suite

Parallel execution of the check suite

Very lengthy check instances pose a big problem when working steady integration (CI) at scale. As initiatives develop in complexity and group measurement, the time required to execute complete check suites can enhance dramatically, resulting in prolonged pipeline execution instances. This not solely delays the supply of recent options and bug fixes, but in addition hampers developer productiveness by forcing them to attend for construct outcomes earlier than continuing with their duties. I’ve skilled pipelines that took as much as 60 minutes to run, solely to fail on the final step, requiring a whole rerun and additional delays. These prolonged cycles can erode developer belief within the CI course of, contribute to frustration, and finally decelerate your entire software program supply cycle. Furthermore, long-running exams can result in useful resource competition, elevated prices due to wasted computing energy, and diminished total effectivity of the event course of.

With parallel check execution in CodeBuild, now you can run your exams concurrently throughout a number of construct compute environments. This characteristic implements a sharding strategy the place every construct node independently executes a subset of your check suite. CodeBuild offers surroundings variables that determine the present node quantity and the entire variety of nodes, that are used to find out which exams every node ought to run. There isn’t any management construct node or coordination between nodes at construct time—every node operates independently to execute its assigned portion of your exams.

To allow check splitting, configure the batch fanout part in your buildspec.xml, specifying the specified parallelism degree and different related parameters. Moreover, use the codebuild-tests-run utility in your construct step, together with the suitable check instructions and the chosen splitting methodology.

The exams are break up primarily based on the sharding technique you specify. codebuild-tests-run provides two sharding methods:

• Equal-distribution. This technique kinds check information alphabetically and distributes them in chunks equally throughout parallel check environments. Adjustments within the names or amount of check information may reassign information throughout shards.
• Stability. This technique fixes the distribution of exams throughout shards by utilizing a constant hashing algorithm. It maintains current file-to-shard assignments when new information are added or eliminated.

CodeBuild helps computerized merging of check stories when working exams in parallel. With computerized check report merging, CodeBuild consolidates exams stories right into a single check abstract, simplifying end result evaluation. The merged report consists of aggregated cross/fail statuses, check durations, and failure particulars, decreasing the necessity for guide report processing. You’ll be able to view the merged leads to the CodeBuild console, retrieve them utilizing the AWS Command Line Interface (AWS CLI), or combine them with different reporting instruments to streamline check evaluation.

Let’s have a look at the way it works
Let me reveal how one can implement parallel testing in a undertaking. For this demo, I created a really fundamental Python undertaking with a whole bunch of exams. To hurry issues up, I requested Amazon Q Developer on the command line to create a undertaking and 1,800 check instances. Every check case is in a separate file and takes one second to finish. Operating all exams in a sequence requires half-hour, excluding the time to provision the surroundings.

On this demo, I run the check suite on ten compute environments in parallel and measure how lengthy it takes to run the suite.

To take action, I added a buildspec.yml file to my undertaking.

model: 0.2

batch:
  fast-fail: false
  build-fanout:
    parallelism: 10 # ten runtime environments 
    ignore-failure: false

phases:
  set up:
    instructions:
      - echo 'Putting in Python dependencies'
      - dnf set up -y python3 python3-pip
      - pip3 set up --upgrade pip
      - pip3 set up pytest
  construct:
    instructions:
      - echo 'Operating Python Exams'
      - |
         codebuild-tests-run 
          --test-command 'python -m pytest --junitxml=report/test_report.xml' 
          --files-search "codebuild-glob-search 'exams/test_*.py'" 
          --sharding-strategy 'equal-distribution'
  post_build:
    instructions:
      - echo "Take a look at execution accomplished"

stories:
  pytest_reports:
    information:
      - "*.xml"
    base-directory: "report"
    file-format: JUNITXML 

There are three components to spotlight within the YAML file.

First, there’s a build-fanout part below batch. The parallelism command tells CodeBuild what number of check environments to run in parallel. The ignore-failure command signifies if failure in any of the fanout construct duties could be ignored.

Second, I exploit the pre-installed codebuild-tests-run command to run my exams.

This command receives the entire listing of check information and decides which of the exams should be run on the present node.

• Use the sharding-strategy argument to decide on between equally distributed or secure distribution, as I defined earlier.
• Use the files-search argument to cross all of the information which are candidates for a run. We advocate to make use of the supplied codebuild-glob-search command for efficiency causes, however any file search software, corresponding to discover(1), will work.
• I cross the precise check command to run on the shard with the test-command argument.

Lastly, the stories part instructs CodeBuild to gather and merge the check stories on every node.

Then, I open the CodeBuild console to create a undertaking and a batch construct configuration for this undertaking. There’s nothing new right here, so I’ll spare you the main points. The documentation has all the main points to get you began.  Parallel testing works on batch builds. Be sure to configure your undertaking to run in batch.

Now, I’m able to set off an execution of the check suite. I can commit new code on my GitHub repository or set off the construct within the console.

After a couple of minutes, I see a standing report of the completely different steps of the construct; with a standing for every check surroundings or shard.

When the check is full, I choose the Studies tab to entry the merged check stories.

The Studies part aggregates all check information from all shards and retains the historical past for all builds. I choose my most up-to-date construct within the Report historical past part to entry the detailed report.

As anticipated, I can see the aggregated and the person standing for every of my 1,800 check instances. On this demo, they’re all passing, and the report is inexperienced.

The 1,800 exams of the demo undertaking take one second every to finish. After I run this check suite sequentially, it took 35 minutes to finish. After I run the check suite in parallel on ten compute environments, it took 6 minutes to finish, together with the time to provision the environments. The parallel run took 17.9 % of the time of the sequential run. Precise numbers will fluctuate along with your initiatives.

Further issues to know
This new functionality is appropriate with all testing frameworks. The documentation consists of examples for Django, Elixir, Go, Java (Maven), Javascript (Jest), Kotlin, PHPUnit, Pytest, Ruby (Cucumber), and Ruby (RSpec).

For check frameworks that don’t settle for space-separated lists, the codebuild-tests-run CLI offers a versatile various via the CODEBUILD_CURRENT_SHARD_FILES surroundings variable. This variable incorporates a newline-separated listing of check file paths for the present construct shard. You should use it to adapt to completely different check framework necessities and format check file names.

You’ll be able to additional customise how exams are break up throughout environments by writing your personal sharding script and utilizing the CODEBUILD_BATCH_BUILD_IDENTIFIER surroundings variable, which is robotically set in every construct. You should use this method to implement framework-specific parallelization or optimization.

Pricing and availability
With parallel check execution, now you can full your check suites in a fraction of the time beforehand required, accelerating your improvement cycle and enhancing your group’s productiveness.

Parallel check execution is on the market on all three compute modes supplied by CodeBuild: on-demand, reserved capability, and AWS Lambda compute.

This functionality is on the market at present in all AWS Areas the place CodeBuild is obtainable, with no further price past the usual CodeBuild pricing for the compute assets used.

I invite you to strive parallel check execution in CodeBuild at present. Go to the AWS CodeBuild documentation to be taught extra and get began with parallelizing your exams.

— seb

PS: Right here’s the immediate I used to create the demo software and its check suite: “I’m writing a weblog put up to announce codebuild parallel testing. Write a quite simple python app that has a whole bunch of exams, every check in a separate check file. Every check takes one second to finish.”

How is the Information Weblog doing? Take this 1 minute survey!

(This survey is hosted by an exterior firm. AWS handles your data as described within the AWS Privateness Discover. AWS will personal the info gathered by way of this survey and won’t share the data collected with survey respondents.)