How to execute parallel tasks

In this guide we will demonstrate how to perform tasks in parallel.

Introduction

Resources with no dependencies can often be deployed in parallel. This can drastically speed up our deployments, especially if we have a lot of time-consuming tasks. In this demo will we look at how to do this using Attini.

How to use

To execute tasks in parallel you have to use the “Parallel” type. This is standard StepFunction type, but it can be written with the Attini simplified syntax.

Type: Parallel
Branches:
  - #Branch 1
    - Step1 
  - #Branch 2
    - Step2

As you can se above, Branches contains a list of lists. Each list can contain a number of steps that will be executed in the branch. The parallel step will be completed when all the branches have finished successfully.

Normally when using StepFunctions parallel type, the output of the step will be a list containing the outputs of all the branches. This makes querying the result difficult for later steps, as well as duplicating a lot of data unnecessarily. The “AttiniMergeOutput” step handles this by merging all the branches in to a single object. When using the simplified syntax the merge step will be added automatically after every parallel step.

Demo

In this demo we will deploy two CloudFormation stacks in parallel. Each stack contains a lambda that will return the name of the step that deployed it. We will then use the “AttiniLambdaInvoke” step to call both the lambdas.

Our project has the following file structure:

.
├── attini-config.yaml
├── deployment-plan.yaml
└── lambda.yaml

Complete examples of all the files can be found at the bottom of this page. But the relevant part for this demo is the deployment plan, which looks like this:

DeploymentPlan:
  - Name: deploy-my-lambdas
    Type: Parallel
    Branches:
      -
        - Name: DeployLambdaNumber1
          Type: AttiniCfn
          Properties:
            StackName: lambda-nr-1
            Template: /lambda.yaml
            Parameters:
              StepName.$ : $$.State.Name
        - Name: InvokeLambdaNumber1
          Type: AttiniLambdaInvoke
          Parameters:
            FunctionName.$: $.output.DeployLambdaNumber1.FunctionName
      -
        - Name: DeployLambdaNumber2
          Type: AttiniCfn
          Properties:
            StackName: lambda-nr-2
            Template: /lambda.yaml
            Parameters:
              StepName.$: $$.State.Name
        - Name: InvokeLambdaNumber2
          Type: AttiniLambdaInvoke
          Parameters:
            FunctionName.$: $.output.DeployLambdaNumber2.FunctionName

As you can see each parallel branch has two steps, one that deploys the lambda and one that calls it. Let’s deploy the distribution running the attini deploy run command from the root of the project:

attini deploy run .

Once the deployment has finished we got the following output:

example parallel

As we can see both lambdas where deployed and called. We can also see that the merge step was added in the end. The merge step merged the output of all the branches for us in to a single object, making it easier to query the result of the different branches.

Example files

deployment-plan.yaml
AWSTemplateFormatVersion: "2010-09-09"
Transform:
  - AttiniDeploymentPlan
  - AWS::Serverless-2016-10-31

Resources:
  ParallelDemo:
    Type: Attini::Deploy::DeploymentPlan
    Properties:
      DeploymentPlan:
        - Name: deploy-my-lambdas
          Type: Parallel
          Branches:
            -
              - Name: DeployLambdaNumber1
                Type: AttiniCfn
                Properties:
                  StackName: lambda-nr-1
                  Template: /lambda.yaml
                  Parameters:
                    StepName.$ : $$.State.Name
              - Name: InvokeLambdaNumber1
                Type: AttiniLambdaInvoke
                Parameters:
                  FunctionName.$: $.output.DeployLambdaNumber1.FunctionName
            -
              - Name: DeployLambdaNumber2
                Type: AttiniCfn
                Properties:
                  StackName: lambda-nr-2
                  Template: /lambda.yaml
                  Parameters:
                    StepName.$: $$.State.Name
              - Name: InvokeLambdaNumber2
                Type: AttiniLambdaInvoke
                Parameters:
                  FunctionName.$: $.output.DeployLambdaNumber2.FunctionName
attini-config.yaml
distributionName: attini-parallel-demo
initDeployConfig:
  template: deployment-plan.yaml 
  stackName: ${environment}-${distributionName}-deployment-plan 

package:
  prePackage:
    commands: 
      - attini configure set-dist-id --random

lambda.yaml

AWSTemplateFormatVersion: 2010-09-09
Transform: AWS::Serverless-2016-10-31

Parameters:
  StepName:
    Type: String

Resources:

  DeployNameLambda:
    Type: AWS::Serverless::Function
    Properties:
      Description:  Lambda that returns the name of the step that deployed it
      Environment:
        Variables:
          STEP_NAME: !Ref StepName
      InlineCode: |
          import os
          def lambda_handler(event, context):
            stepName = os.environ["STEP_NAME"]
            return f"I was deployed by step with name: {stepName}"          
      Handler: index.lambda_handler
      Runtime: python3.9


  DeployNameLogGroup:
    Type: AWS::Logs::LogGroup
    Properties:
      LogGroupName: !Sub /aws/lambda/${DeployNameLambda}
      RetentionInDays: 30

Outputs:
  FunctionName:
    Value: !Ref DeployNameLambda