Azure pipeline as a core part of Azure DevOps, it allows for the creation of CI ( Continuous Integration) pipeline in a declarative way using YAML documents, it is also called build pipelines. Since last Build 2019, this capability is also extending to CD ( Continuous Delivery ) Pipelines which is also known as Release Pipelines.  More than expected is now it is possible to define multi-stage pipelines-as-code for both Continuous Integration and Continuous Delivery with the same YAML definition file. 

Since Github can be easily integrated with Azure DevOps nowadays,  you can not only build your CI/CD pipeline based on your source code on Github, but also even Mapping your GitHub repository permissions with Azure DevOps.

Solution Overview

I am writing this blog to explain how to use Azure CI/CD pipelines to provide an end-to-end automation experience to users when deploying a node.js application via Azure DevOps. Our solution will look like the following diagram : 

         End-to-End CI/CD architecture diagram 

Getting started

Before getting started, you have to prepare a couple of things : 

Step 1: Get an Azure subscription

You need an Azure account: Create an Azure account by browsing to https://azure.microsoft.com/en-us/free/ or claim your MSDN benefits to get a visual studio subscription. 

Step 2 : Get an Azure DevOps account and create organisation and project. You can check Quickstart: Create an organization or project collection to know more details.

You’ll need to activate the ‘ multi-stage pipeline’ from the settings of preview feature. 

  Activate the multi-stage pipelines preview feature

Step 3 : Get source from cloudmelon’s Github repository ( using the following URL ), fork it and use your own branch for our lab here : https://cloudmelon.github.io/oss-cicd-devops/

Provision infrastructure via ARM template

One of the most important part of end-to-end automation start from automating the infrastructure provision process,  by implementing Infrastructure as code (IaC) mindsets, we can use either ARM or Terraform to provision our infrastructure running in Azure.

In this solution, we’re going to provision two App services Plan and host web app on each of them. In the enterprise environment, due to budget and performance concerns, usually, we deploy Development and Production environment separately. On the top of the solution, we can also use full stack monitoring solutions to gain more visibility of Azure resources which we’ll get to there later on. 

Where DevOps magic happens: defining service connections

There are so many discussions around automation and DevOps going on over past a few years. From my point of view, one of the most charming features of DevOps is you can deploy and automate the whole process as long as you have the right permission. In our example, you only need to replace the service connections then you can get this solution up and running in your subscription.  You only need to go to Project settings and add a new service connection as shown in the following : 

    Create service connections

Trigger and variable definition

Basically, Continuous integration (CI) triggers in Azure DevOps will set off a build to run when a push is made to the specified branches ( by setting path filters ) or a specified tag is pushed. Below is an example of trigger defined in Yaml:

trigger:
- master

In Azure DevOps, you can use a variable in Yaml definition or using Variable group and then link up to your CI or CD definition. Below is an example of yaml-defined variables :

variables:

  # Agent VM image name
  vmImageName: 'ubuntu-latest'
  demorg: 'melon-cicd-rg'
  subscription : '(please replace it by your own service connection name)'
  webappname : 'melonsamplewebapp'

Define your CI pipeline

In your CI pipeline, you need to define as the following :

- stage: Build
  displayName: Build stage
  jobs:  
  - job: Build
    displayName: Build
    pool:
      vmImage: $(vmImageName)
    steps:
    - task: AzureCLI@1
      displayName: 'Azure CLI '
      inputs:
        azureSubscription: $(subscription)
        scriptLocation: inlineScript
        inlineScript: 'az group create --location northeurope --name $(demorg)'

    - task: AzureResourceGroupDeployment@2
      displayName: 'Azure Deployment:Create Or Update Resource Group action on $(demorg)'
      inputs:
        azureSubscription: $(subscription)
        resourceGroupName: '$(demorg)'
        location: 'North Europe'
        templateLocation: 'Linked artifact'
        csmFile: 'iac/webapp.json'
        csmParametersFile: 'iac/webapp.parameters.json'
        deploymentMode: 'Incremental'

    - task: NodeTool@0
      inputs:
        versionSpec: '10.x'
      displayName: 'Install Node.js'

    - script: |
        npm install
        npm run build --if-present
        # npm run test --if-present
      displayName: 'npm install, build and test'


    - task: CopyFiles@2
      displayName: 'Copy Files to: $(Build.ArtifactStagingDirectory)/$(webappname)'
      inputs:
        SourceFolder: '$(system.defaultworkingdirectory)'
        TargetFolder: '$(Build.ArtifactStagingDirectory)/$(webappname)'

    - task: ArchiveFiles@2
      displayName: '$(webappname) Archive'
      inputs:
        rootFolderOrFile: '$(Build.ArtifactStagingDirectory)/$(webappname)'
        includeRootFolder: false
        archiveType: zip
        replaceExistingArchive: true
        archiveFile: '$(Build.ArtifactStagingDirectory)/$(webappname).zip'


    - task: PublishPipelineArtifact@0
      displayName: 'PublishPipelineArtifact: drop'
      inputs:
        targetPath: '$(Build.ArtifactStagingDirectory)/$(webappname).zip'

Define your CD pipeline

Now yaml-defined multi-stage CD pipeline is also supported in Azure DevOps, if you’re not comfortable with yaml definition, you can also check Define your multi-stage continuous deployment (CD) pipeline to know more about multi-stage pipeline. 

Please make sure you define different stages, dependencies and deployment condition :

Dev stage :

- stage: Dev
  displayName: Dev stage
  dependsOn: Build
  condition: succeeded('Build')

  jobs:
  - deployment: Dev
    displayName: Dev
    environment: 'development'

    pool: 
      vmImage: $(vmImageName)

    strategy:
      runOnce:
        deploy:
          steps:
          - task: DownloadPipelineArtifact@1
            displayName: 'Download Pipeline Artifact'
            inputs:
              buildType: 'current'

          - task: AzureWebApp@1
            inputs:
              azureSubscription: '$(subscription)'
              appType: 'webApp'
              appName: '$(webappname)'
              package: '$(System.ArtifactsDirectory)/drop/$(webappname).zip'
              customWebConfig: '-Handler iisnode -NodeStartFile index.js -appType node'
              deploymentMethod: 'zipDeploy'

Prod stage :

- stage: Prod
  displayName: Prod stage
  dependsOn: Dev
  condition: succeeded('Dev')

  jobs:
  - deployment: Prod
    displayName: Prod
    environment: 'production'
    pool: 
      vmImage: $(vmImageName)

    strategy:
      runOnce:
        deploy:
          steps:
          - task: DownloadPipelineArtifact@1
            displayName: 'Download Pipeline Artifact'
            inputs:
              buildType: 'current'


          - task: AzureWebApp@1
            inputs:
              azureSubscription: '$(subscription)'
              appType: 'webApp'
              appName: '$(prodwebappname)'
              package: '$(System.ArtifactsDirectory)/drop/$(webappname).zip'
              customWebConfig: '-Handler iisnode -NodeStartFile index.js -appType node'
              deploymentMethod: 'zipDeploy'

Environments 

The environment is a new feature of Azure DevOps pipeline, it represents a collection of resources such as namespaces within Kubernetes clusters or Azure Web Apps, basically, anything is targeted by deployments from a pipeline. In our example environments include Development and Production environment.

Environments in Azure DevOps

Art of possible: Full stack, end-to-end visibility Azure unified monitoring solution 

Since the end of 2018, Azure Log Analytics and Azure Application Insights were available as integrated features within Azure Monitor,  it is a pretty cool path which shows the possibility to provide administrators and DevOps engineers full stack unified monitoring solution, you can also check End-to-end monitoring solutions in Azure for Apps and Infrastructure to have more details. 

Up & Running 

You can check your run history and different stages in Azure DevOps.

 Deploying multi-stage pipelines in Azure DevOps

Perspectives 

The value of end-to-end automation is to cut down time-to-market and to boot your business performance. The scenarios are not only on OSS or .Net application deployment but also make senses on API economy as well as microservices productivities. However, one of the biggest challenges of this approach is about security and compliance, where DevSecOps comes out, you can find more information about Secure DevOps here. Let’s stay tuned! 

Originally published on 8th July 2019 on https://buildazure.com/end-end-ci-cd-automation-using-azure-devops-unified-yaml-defined-pipelines/ by Melony Qin.