Performance Benchmark

Result

Unit is in seconds.

Note: Attini is not intended to replace the above technologies, just extend them. Attini is designed to speed up development, decrease cost, enable least privilege configuration, make it easier to manage multiple environments, integrate different CI/CD tools, manage backups and restore processes and other hygiene factors into your DevOps workflow. The intention is to use Attini directly from a workstation when doing development, then have ex GitHub Actions or GitLab Runners deploy Attini distributions to acceptance, staging or production environments.

Scenario

We want to know how fast our tooling lets us iterate over our codebase when working with cloud systems. To do this we set up a simple pipeline with Attini, AWS CodePipeline, GitHub Actions and GitLab Runners and measure the time it takes to apply a small change.

We will do all the tests in AWS Ireland region (eu-west-1).

Test steps

We will upload the code from a Cloud9 instance in AWS Irland (eu-west-1). This way network latency will have a minimal impact and enable anyone to reproduce the result.

We run a 3-second sleep to mock a build or script. We are using a sleep method and not an actual build script because we don’t want the size of the container, software dependencies etc to influence the result.

We will update an AWS::SSM::Parameter resource using CloudFormation. This is a quick update, but it will force the stack into UPDATE_IN_PROGRESS. We do it this way because we are not testing CloudFormation, we are testing the pipeline that manages CloudFormation.

We also added 2 steps with a “No change” update on CloudFormation stacks. This is because we rarely update every CloudFormation stack in a pipeline, but we still need to wait for the “No change” steps to pass.

Implentations

Attini

Attini uses the Attini Runner to run the bash command and the AttiniCfn type to deploy CloudFourmation.

Why was Attini so fast?

Warm starts

The Attini Runner is a container built on top of AWS Fargate and it stays warm between executions. When a cold start occurs, the AttiniRunnerJob usually takes 30 to 45 sec longer than shown in the data which puts the execution time at about 1 minute, which is still faster than the alternatives.

Cold starts can largely be avoided by increasing the IdleTimeToLive configuration, but this is a “cost vs performance” decision. If you set the IdleTimeToLive to 12 hours with the default container size, you will only have one cold start per workday, costing roughly 0.15 USD daily.

Event-driven design

The AttiniCfn step is event-driven, while the other tools use often use polling patterns. Using polling is slow, it puts a strain on API limits and increases the costs of any log analytics tool you might use.

Deploymet Plan code

DeploymentPlan:
  Type: Attini::Deploy::DeploymentPlan
  Properties:
    DeploymentPlan:
      - Name: RunBuild
        Type: AttiniRunnerJob
        Properties:
          Runner: Runner
          Commands:
            - sleep 3
      - Name: Parameter1
        Type: AttiniCfn
        Properties:
          StackName: !Sub ${Env}-parameter-1
          Template: /changed-ssm-parameter.yaml
      - Name: Parameter2
        Type: AttiniCfn
        Properties:
          StackName: !Sub ${Env}-parameter-2
          Template: /unchanged-ssm-parameter.yaml
      - Name: Parameter3
        Type: AttiniCfn
        Properties:
          StackName: !Sub ${Env}-parameter-3
          Template: /unchanged-ssm-parameter.yaml

AWS CodePipeline

AWS CodePipeline used AWS CodeBuild to run the bash command and the AWS CloudFormation Action provider to deploy CloudFormation.

The source was a ZIP archive on S3.

The CodePipeline was manually configured so there is no code example.

What took time?

The CloudFormation Action provider was pretty fast, but the CodeBuild startup took some time.

GitHub Action

Github Actions used the native run step to execute the bash command, and action aws-actions/aws-cloudformation-github-deploy@v1 to deploy CloudFormation.

What took time?

GitHub Actions native run command was very quick but the aws-actions/aws-cloudformation-github-deploy@v1 action was quite slow.

Worth noting is that the native run command does not allow you to use a custom image, so if you need any custom software, you would need to install it at the beginning of every run.

Github workflow code

on:
  push:
    branches:
      - main

name: Benchmark

jobs:
  deploy:
    name: Deploy
    runs-on: ubuntu-latest

    steps:
    - name: checkout repo
      uses: actions/checkout@v2.3.4
    - name: Configure AWS Credentials
      uses: aws-actions/configure-aws-credentials@v1
      with:
        aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
        aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
        aws-region: eu-west-1

    - name: RunBuild
      run: sleep 3

    - name: Parameter1
      uses: aws-actions/aws-cloudformation-github-deploy@v1
      with:
        name: benchmark-parameter-1
        template: changed-ssm-parameter.yaml
        no-fail-on-empty-changeset: "1"

    - name: Parameter2
      uses: aws-actions/aws-cloudformation-github-deploy@v1
      with:
        name: benchmark-parameter-2
        template: unchanged-ssm-parameter.yaml
        no-fail-on-empty-changeset: "1"

    - name: Parameter3
      uses: aws-actions/aws-cloudformation-github-deploy@v1
      with:
        name: benchmark-parameter-1
        template: unchanged-ssm-parameter.yaml
        no-fail-on-empty-changeset: "1"

GitLab Runner

The GitLab Runner does not have a native CloudFormation integration, so we used the AWS CLI to run the aws cloudformation deploy command. GitLab Runner default image did not have the AWS CLI installed so we provided our own docker image with the AWS CLI installed, hosted in the GitLab Container repository.

In a way, this is an unfair comparison considering it requires more prep work (maintaining your own image) than the other technologies and the argument could be made that we should just use the standard GitLab Image and install the AWS CLI at the beginning of every run. But the goal of the benchmark is to get compare the technologies in a way that best represents a real-life deployment, and we believe that most IT professionals would build a custom image in this scenario.

What took time?

The deployment using the AWS CLI was quite fast. However, pulling the image was slow. This meant that the result depended a lot on the image size which made it hard to do a fair benchmark. If we used a bare-bone image with just the AWS CLI installed (289 MB), we could get it down to about 80 - 85 seconds which is better than GitHub and AWS CodePiepline. However, that would not be fair because the other technologies used standard images that are quite generous when it comes to pre-installed software.

So to make it a fair comparison, we also installed some common software like nodejs, python3, typescript, AWS CDK, AWS SAM and some other common utils and libs. The image ended up being 593 MB which is still smaller than other standard images but considering that you would probably have a lighter image if you are customizing it, we think it’s a fair test.

Gitlab code

image: registry.gitlab.com/attini-dev/benchmark

stages:
  - Deploy

variables:
  AWS_REGION: eu-west-1
  AWS_ACCOUNT: "111111111111"

deploy:
  stage: Deploy
  before_script:
    - echo "Assuming IAM Role"
    - >
        export $(printf "AWS_ACCESS_KEY_ID=%s AWS_SECRET_ACCESS_KEY=%s AWS_SESSION_TOKEN=%s"
        $(aws sts assume-role-with-web-identity
        --role-arn "arn:aws:iam::${AWS_ACCOUNT}:role/Benchmark"
        --role-session-name "Benchmark-${CI_PROJECT_ID}-${CI_PIPELINE_ID}"
        --web-identity-token $CI_JOB_JWT_V2
        --query 'Credentials.[AccessKeyId,SecretAccessKey,SessionToken]'
        --output text))        
  script:
    - aws cloudformation deploy --template-file ssm-parameter.yaml --stack-name bench-hello-world-parameter-1 --no-fail-on-empty-changeset
    - aws cloudformation deploy --template-file ssm-parameter-old.yaml --stack-name bench-hello-world-parameter-2 --no-fail-on-empty-changeset
    - aws cloudformation deploy --template-file ssm-parameter-old.yaml --stack-name bench-hello-world-parameter-3 --no-fail-on-empty-changeset