Deploying ARM Templates using Visual Studio Team Services

If you are running your applications in Azure, and in particular on PaaS, you need to take a look ARM templates as a way to manage your environments. ARM templates let’s you define and deploy your entire environment using JSON files that you store together with the rest of your source code. The deployment of ARM templates are idempotent, meaning that you can run them many times and it will always produce the same result.

Image result for azure ARM templates

In this post, I will how you how to deploy ARM templates together with your application using Visual Studio Team Services. As you will see, I will not use the out of the box task for doing this, since it has some limitations. Instead we will use a PowerShell script to eexecute the deployment of an ARM template.

The overall steps are:

Defining our ARM template for our environment.
Tokenize the ARM template parameters file
Create a PowerShell script that deploys the ARM template
Deploy everything from a VSTS release definition.

Let’s get started with the ARM template.

ARM Template

In this case, I will deploy an ARM template consisting of a Azure web app, a SQL Server + database and a Redis Cache. The web app and sql resources are easy to deploy, since we can supply all the input from my release definition.
With the Redis cache however, Azure Resource Manager will create some information (such as the primarykey) as part of the deployment, which means we need to read this information from the output of the ARM template deployment.

Here is the outline of our ARM template:

Note the outputs section that is selected above, here we define what output we want to capture once the reource group has been deployed. In this case, I have defined three output variables:

redis_host
The fully qualified edish host name
redis_port
The secure port that will be used to communicate with the cache
redis_primatykey
The access key that we will use to authenticate

Since our web application will communicate with the Redis cache, we need to fetch this information from the ARM template deployment and store them in our web.cofig file. You will see later on how this can be done.

Learn more about authoring ARM templates here: https://docs.microsoft.com/en-us/azure/azure-resource-manager/resource-group-authoring-templates

ARM Template Tokenization

When deploying our template in different environments (dev, test, prod…) we need to supply the information specific to those environment. In VSTS Release Management, the information is stored using environment variables.
A common solution is to tokenize the files that is needed for deployment and then replace these tokens with the corresponding environment variable.

To do this, we add a separate parameters file for the template that contains all the parameters but all the values are replaces with tokens:

{
    “$schema”: “http://schema.management.azure.com/schemas/2015-01-01/deploymentParameters.json#”,
    “contentVersion”: “1.0.0.0”,
“parameters”: {
    “hostingPlanName”: {
      “value”: “__HOSTINGPLANNAME__”
    },
    “administratorLogin”: {
      “value”: “__ADMINISTRATORLOGIN__”
    },
    “administratorLoginPassword”: {
      “value”: “__ADMINISTRATORLOGINPASSWORD__”
    },
    “databaseName”: {
      “value”: “__DATABASENAME__”
    },
    “webSiteName”: {
      “value”: “__WEBAPPNAME__”
    },
    “sqlServerName”: {
      “value”: “__SQLSERVERNAME__”
    },
    “dictionaryName”: {
      “value”: “__DATABASENAMEDICTIONARY__”
    },
    “extranetName”: {
      “value”: “__DATABASENAMEEXTRANET__”
    },
    “instanceCacheName”: {
      “value”: “__INSTANCECACHENAME__”
    }

}
}

We wil then replace these tokens just before the template is deployed.

PowerShell script

There is an existing task for creating and updating ARM templates, called Azure Resource Group Deployment. This task let’s us point to an existing ARM template and the corresponding parameter file.

Here is an example how how this task is typically used:

The problem with this task is that it has very limited support for output parameters. As you can see in the image above, you can map a variable to the output called Resource Group. Unfortunately there is an assumption that the resource group that you are creating contains virtual machines. If you execute this task with an ARM template containing for example an Azure Web App you will get the following error when trying to map the output to a variable:

2017-01-23T09:09:49.8436157Z ##[error]The ‘Get-AzureVM’ command was found in the module ‘Azure’, but the module could not be loaded. For more information, run ‘Import-Module Azure’.

So, to be able to read our output values we need to use PowerShell instead, which is arguably a better choice anyway since it allows you to run and test the deployment locally, saving you a lot of time.

When we create an Azure Resource Group project in Visual Studio, we get a PowerShell script that we can use as a starting point.

Most part of this script handles the case where we need to upload artifacts as part of the resource group deployment. In this case we don’t need this, we deploy all our artifacts from RM after the resource group has been deployed.

Here is our PowerShell script that we will use to deploy the template:

#Requires -Version 3.0
#Requires -Module AzureRM.Resources
#Requires -Module Azure.Storage

Param(
    [string] [Parameter(Mandatory=$true)] $ResourceGroupLocation,
    [string] [Parameter(Mandatory=$true)] $ResourceGroupName,
    [string] [Parameter(Mandatory=$true)] $TemplateFile,
    [string] [Parameter(Mandatory=$true)] $TemplateParametersFile
)

Import-Module Azure -ErrorAction SilentlyContinue

try {
[Microsoft.Azure.Common.Authentication.AzureSession]::ClientFactory.AddUserAgent(“VSAzureTools-$UI$($host.name)”.replace(” “,”_”), “2.9”)
} catch { }

Set-StrictMode -Version 3

$TemplateFile = [System.IO.Path]::GetFullPath([System.IO.Path]::Combine($PSScriptRoot, $TemplateFile))
$TemplateParametersFile = [System.IO.Path]::GetFullPath([System.IO.Path]::Combine($PSScriptRoot, $TemplateParametersFile))

# Create or update the resource group using the specified template file and template parameters file
New-AzureRmResourceGroup -Name $ResourceGroupName -Location $ResourceGroupLocation -Verbose -Force -ErrorAction Stop

$output = (New-AzureRmResourceGroupDeployment -Name ((Get-ChildItem $TemplateFile).BaseName + ‘-‘ + ((Get-Date).ToUniversalTime()).ToString(‘MMdd-HHmm’)) `
                                   -ResourceGroupName $ResourceGroupName `
                                   -TemplateFile $TemplateFile `
                                   -TemplateParameterFile $TemplateParametersFile -Force -Verbose)

Write-Output (“##vso[task.setvariable variable=REDISSERVER]” + $output.Outputs[‘redis_host’].Value)
Write-Output (“##vso[task.setvariable variable=REDISPORT]” + $output.Outputs[‘redis_port’].Value)
Write-Output (“##vso[task.setvariable variable=REDISPASSWORD;issecret=true]” + $output.Outputs[‘redis_primarykey’].Value)

The special part of this script is the last three lines. Here, we read the output variables that we defined in the ARM template and then we use one of the VSTS logging commands to map these into variables that we can use in our release definition.

The syntax of the SetVariable logging command is ##vso[task.setvariable variable=NAME]<VARIABLEVALUE>.

Note: You can read more about these commands at https://github.com/Microsoft/vsts-tasks/blob/master/docs/authoring/commands.md

Release Definition

Finally we can put all of this together by creating a release definition that deploys the ARM template.

Note: You will of course need to create a build definition that packages your scripts, ARM templates and deployment artifacts. I won’t show this here, but just reference the outputs from an existing build definition.

Here is what the release definition will look like:

Let’s walk through the steps:

Replace tokens
Here we replace the tokens in our parameters.json file that we definied earlier. There are several tasks in the marketplace for doing token replacement, I’m using the one from Guillaume Rouchon (https://github.com/qetza/vsts-replacetokens-task#readme)
Deploy Azure environment
Run the PowerShell scipt using the Azure PowerShell task. This task handles the connection to Azure, so we don’t have to think about that.

Here I reference the PowerShell script from the build output artifacts, and also I supply the necessary parameters to the PS script:

Script Arguments
-ResourceGroupLocation “$(resourceGroupLocation)” -ResourceGroupName $(resourceGroupName) -TemplateFile “$(System.DefaultWorkingDirectory)/SampleApp.CI/environment/templates/sampleapp.json” -TemplateParametersFile “$(System.DefaultWorkingDirectory)/SampleApp.CI/environment/templates/sampleapp.parameters.json”

Replace tokens
Now we need to update the tokens in our SetParameters file, that is used by web deploy. It is important that we run this task after running the deploy azure enviroment script, since we need the output variables from the resource group deployment. Remember, these variables are now available as environment variables, so they will be inserted in the same way as the variables that we have defined manually.
Deploy Web app + Deploy SQL Database
These steps just performs a simple deployment of an Azure Web App and a SQL dacpac deployment.

That’s it, happy deployment! Smile

Using Web Deploy in Visual Studio Team Services Release Management

This post does not really cover something new, but since I find myself explain this to people now and then, I thought that I’d write a quick post on the subject.

So, we want to create a web deploy package as part of our automated build, and then take this package and deploy it to multiple environments, where each environment can have different configuration settings, using VSTS Release Management. Since we only want to build our package once, we have to apply the environment specific settings at deployment time, which means we will use Web Deploy parameters.

Here are the overall steps needed:

– Create a parameters.xml file in your web project
– Create a publish profile for the web deploy package
– Set up a VSTS build creates the web deploy package, and uploads the package to the server
– Create a Release definition in VSTS that consumes the web deploy package
– In each RM environment, replaces the tokens in the SetParameters file

Let’s run through these steps in detail:

Create a parameters.xml file

As you will see later on, a publish profile contains configurable settings for the web site name and any connection strings,that will end up in the *.SetParameters.xml file that is used when at deployment time. But in order for other configuration settings, like appSettings, to end up in this file, you need to define these settings. This is done by creating a file called parameters.xml in the root of your web application.

Tip: A fellow MVP, Richard Fennell, has created a nifty Visual Studio extension that simplifies the process of creating the parameters.xml file. It will look at your web.config file and the create a parameters.xml file with all the settings that it finds.

In this case, I have three application settings in the web.config file, so I end up with this parameters.xml file. Note that I have set the defaultvalue attribute for all parameters to __TOKEN__. These are the configuration values that will end up in the MyApp.SetParameters.xml file, together with the web deployment package. We will replaced these values at deployment time, by a task in our release definition.

<parameters>
      
      <parameter name="IsDevelopment" description="Description for IsDevelopment" defaultvalue="__ISDEVELOPMENT__" tags="">
      <parameterentry kind="XmlFile" scope="\\web.config$" match="/configuration/applicationSettings/MyApp.Properties.Settings/setting[@name='IsDevelopment']/value/text()" />
      </parameter>
        
      <parameter name="WebApiBaseUrl" description="Description for WebApiBaseUrl" defaultvalue="__WEBAPIBASEURL__" tags="">
      <parameterentry kind="XmlFile" scope="\\web.config$" match="/configuration/applicationSettings/MyApp.Properties.Settings/setting[@name='WebApiBaseUrl']/value/text()" />
      </parameter>  
      
      <parameter name="SearchFilterDelta" description="Description for SearchFilterDelta" defaultvalue="__SEARCHFILTERDELTA__" tags="">
      <parameterentry kind="XmlFile" scope="\\web.config$" match="/configuration/applicationSettings/MyApp.Properties.Settings/setting[@name='SearchFilterDelta']/value/text()" />
      </parameter>
        
</parameters>

Creating a Publish Profile

Now, let’s create a publish profile that define how the web deployment package should be created. Right-click on the web application project and then select Publish. Then select the Custom option:

Since the publish profile will be used to create a web deployment package, I like to call it CreatePackage (but you are of course free to call it whatever you want)

On the Connection tab, select Web Deploy Package as the publish method, then give the generated package a name (including .zip).

As the Web Site name, we enter a tokenized value __WEBSITE__. This token will also end up in the MyApp.SetParameters.xml file.

Save the publish profile and commit and push your changes. Now it’s time to create a build definition.

Create a Build Definition that generates a web deploy package

I won’t go through all the details of creating a build definition in VSTS, but just focus on the relevant parts for this blog post.

To generate a web deploy package, we need to pass some magic MSBuild parameters as part of the Visual Studio build task. Since we have a publish profile that contains our settings, we need to refer to this file. We also want to specify where the resulting files should be placed.

Enter the following string in the MSBuild Arguments field:

/p:DeployOnBuild=true /p:PublishProfile=CreatePackage /p:PackageLocation=$(build.stagingDirectory)

DeployOnBuild=true is required to trigger the web deployment publishing process, and the we use the PackageLocation property to specify that the output should be places in the staging directory of the build. This will make it easy to upload the artifacts at the end of the build, like so:

This will generate an artifact called drop in the build that contains all files needed to deploy this application using MSDeploy:

As you can see, we have all the generated web deploy files here. We will use three of them:

MyApp.zip – The web deploy package

MyApp.SetParameters.xml – The parameterization file that contains our tokenized parameters

MyApp.Deploy.cmd – A command file that simplifies running MSDeploy with the correct parameters

Creating a Release Definition

Finally, we will create a release definition that deploys this web deploy package to two different environments, let’s call them Test and Prod. In each environment we need to apply the correct configuration values. To do this, we have to replace the token variables in our MyApp.SetParameters.xml file.

There is no out of the box task to do this currently, but there are already several of them in the Visual Studio Marketplace. Here, I will use the Replace Tokens task from Guillaume Rochon, available at https://marketplace.visualstudio.com/items?itemName=qetza.replacetokens. Install it to your Visual Studio Team Services account, and then the task will be available in the build/release task catalog, in the Utility category:

Each environment in the release definition will just contain two tasks, the first one for the token replacement and the other one for deploying the web deploy package. To do this, we just run the MyApp.deploy.cmd file that was generated by the build. Since the parameters have already been set with the correct values, we can just run this without any extra arguments.

Also, we must specify the values for each variable in the environment. Right click on the environment and the add these variables:

Tip: Create the Test environment first with all variables and tasks. Once it’s done, use the Clone environment feature to create a Prod environment, and then just replace the configuration values

That’s it, now you can run the release and it will deploy your web application with the correct configuration to each environment.

New Book – Continuous Delivery with Visual Studio ALM 2015

With today’s announcement at Microsoft Connect() about the public preview of the next generation of Visual Studio Release Management, it is also time to announce the (imminent) release of a new book that covers among other things this new version of RM.

Me and my fellow ALM MVP Mathias Olausson have been working hard during the last 6 months on this book, using early alpha and beta versions of this brand new version of Visual Studio Release Management. Writing about a changing platform can be rather challenging, and our publisher (Apress) have been very patient with us regarding delays and late changes!

About the book

The book is titled Continuous Delivery with Visual Studio ALM 2015 and is aiming to be a more practical complement to Jez Humble’s seminal Continous Delivery book with a heavy focus of course on how to implement these processes using the Visual Studio ALM platform.

The book discusses the principles and practices around continuous delivery and continuous deployment, including release planning, source control management, build and test automation and deployment pipelines. The book uses a fictive sample application that we use throughout the book as a concrete example on how to go about to implement a continuous delivery workflow on a real application.

We hope that you will find this book useful and valuable!

Abstract

This book is the authoritative source on implementing Continuous Delivery practices using Microsoft’s Visual Studio and TFS 2015. Microsoft MVP authors Mathias Olausson and Jakob Ehn translate the theory behind this methodology and show step by step how to implement Continuous Delivery in a real world environment.

Building good software is challenging. Building high-quality software on a tight schedule can be close to impossible. Continuous Delivery is an agile and iterative technique that enables developers to deliver solid, working software in every iteration. Continuous delivery practices help IT organizations reduce risk and potentially become as nimble, agile, and innovative as startups.

In this book, you’ll learn:

What Continuous Delivery is and how to use it to create better software more efficiently using Visual Studio 2015
How to use Team Foundation Server 2015 and Visual Studio Online to plan, design, and implement powerful and reliable deployment pipelines
Detailed step-by-step instructions for implementing Continuous Delivery on a real project

Table of Content

Chapter 1: Introduction to Continuous Delivery
Chapter 2: Overview of Visual Studio 2015 ALM
Chapter 3: Designing an Application for Continuous Delivery
Chapter 4: Managing the Release Process
Chapter 5: Source Control Management
Chapter 6: PowerShell for Deployment
Chapter 7: Build Automation
Chapter 8: Managing Code Quality
Chapter 9: Continuous Testing
Chapter 10: Building a Deployment Pipeline
Chapter 11: Measure and Learn

Trigger Releases in Visual Studio Release Management

In my previous post, I showed how you can trigger a release in Visual Studio Release Management from a TeamCity build step.

When Visual Studio Release Management 2013 RTM’ed, it came with customized TFS build templates that made it easy to trigger a release from a TFVC or Git build in TFS. These build templates relied on the ReleaseManagementBuild command line client, so it required the VSRM client being installed on the build server.

Then, with the update train of Visual Studio 2013, new functionality such as vNext deployments, support for Powershell DSC etc was added. Together with this a new REST API was introduced that removes the need for a command line application to trigger a release. However, this REST API _only_ works for vNext deployments, and it will most likely never be implemented for agent based deployments. Basically all new functionality in the release management area only works for vNext deployements, so you should be looking to move in that direction.

Also, in VS 2013 Update 4 support was added for connecting to the Release Management service running in Visual Studio Online. With this option, another way of triggering release was added, in which the Release Management service listens for build completion events in TFS which will kick off a release. It is still possible to use the API however, in cases where you don’t run TFS Build.

So, since this is rather confusing at the moment and causes a lot of questions, I decided to summarize the currently available combinations and the options that you have when it comes to triggering a release in VSRM:

Source	Release Management	Deployment Type	Options
VSO Build	VSO	vNext	Automatically (through build completed event)
VSO Build	VSO	Agent-Based	N/A
TFS Build	On-Premise (Update 3)	vNext	REST API
TFS Build	On-Premise (Update 3)	Agent-Based	ReleaseBuildTemplate
3^rd party/share	VSO	vNext	REST API
3^rd party/share	VSO	Agent-Based	N/A
3^rd party/share	On-Premise (Update 3)	vNext	REST API
3^rd party/share	On-Premise (Update 3)	Agent-Based	ReleaseManagementBuild.exe

Note that some of these combinations are not supported at the moment, those are marked with N/A.

Hope that helps

Trigger Visual Studio Release Management vNext from TeamCity

The last couple of updates to Visual Studio has included a lot of new functionality for Visual Studio Release Management. The biggest one is the introduction of so called vNext releases, that leverages Powershell DSC for carrying out the provisioning and deployments of environments and applications.

Also included in Visual Studio 2013 Update 3 was the introduction of a REST API that allows us to both trigger new releases and read back information about them. This API is only available for vNext release templates, and will probably not be implemented for the “old” agentbased deployments.

This REST API opens up a lot of integration possibilities, for example we don’t have to use TFS Build to automatically trigger a release. In this post, I will show how we can use the popular TeamCity build server from JetBrains to trigger a release as part of the build.

First of all, we need to setup a release template in Visual Studio Release Management. I won’t go through all the details here, the most important once is that the components that we define must use the UNC Path as the source parameter:

Here I have specified a shared folder, \localhostbuildoutput. Beneath this path, VSRM will look for a folder that typically correspond to a build number that we will pass in using the API, as you will see later on.

Next, we create our release template. To be able to trigger a build using the API, make sure that you tick the “Can trigger a release from a Build?” checkbox:

Then we define our deployment sequence, in this case I just have one action for deploying my web site using a PowerShell script:

There is nothing really nothing special here, I refere to the DeployWebSite.ps1 which is a PowerShell DSC script that installs my web site. I have also a DestinationPath variable that is referenced in the DSC script and specifies where the web site should be installed on the server.

Now, to trigger a release of this release template we are going to use a PowerShell script, that we can execute using TeamCity. I have used a sample from Microsoft and modified it to use parameters.

Here is the full script:

param(
    [string]$rmServer,
    [string]$rmUser,
    [string]$rmPassword,
    [string]$rmDomain,
    [string]$releaseDefinition,
    [string]$deploymentPropertyBag
    )

$deploymentPropertyBag = 
$propertyBag = [System.Uri]::EscapeDataString($deploymentPropertyBag)
$exitCode = 0

trap
{
  $e = $error[0].Exception
  $e.Message
  $e.StackTrace
  if ($exitCode -eq 0) { $exitCode = 1 }
}

$scriptName = $MyInvocation.MyCommand.Name
$scriptPath = Split-Path -Parent (Get-Variable MyInvocation -Scope Script).Value.MyCommand.Path

Push-Location $scriptPath  

$orchestratorService = "http://$rmServer/account/releaseManagementService/_apis/releaseManagement/OrchestratorService"

$status = @{
    "2" = "InProgress";
    "3" = "Released";
    "4" = "Stopped";
    "5" = "Rejected";
    "6" = "Abandoned";
}

#For Update3 use api-version=2.0 for Update4 use api-version=3.0.
$uri = "$orchestratorService/InitiateRelease?releaseTemplateName=" + $releaseDefinition + "&deploymentPropertyBag=" + $propertyBag + "&api-version=3.0"

$wc = New-Object System.Net.WebClient
#$wc.UseDefaultCredentials = $true
# rmuser should be part rm users list and he should have permission to trigger the release.
$wc.Credentials = new-object System.Net.NetworkCredential("$rmUser", "$rmPassword", "$rmDomain")

try
{
    $releaseId = $wc.UploadString($uri,"")
    $url = "$orchestratorService/ReleaseStatus?releaseId=$releaseId"
    $releaseStatus = $wc.DownloadString($url)

    Write-Host -NoNewline "`nReleasing ..."

    while($status[$releaseStatus] -eq "InProgress")
    {
        Start-Sleep -s 5
        $releaseStatus = $wc.DownloadString($url)
        Write-Host -NoNewline "."
    }

    " done.`n`nRelease completed with {0} status." -f $status[$releaseStatus]
}
catch [System.Exception]
{
    if ($exitCode -eq 0) { $exitCode = 1 }
    Write-Host "`n$_`n" -ForegroundColor Red
}

if ($exitCode -eq 0)
{
  "`nThe script completed successfully.`n"
}
else
{
  $err = "Exiting with error: " + $exitCode + "`n"
  Write-Host $err -ForegroundColor Red
}

Pop-Location

exit $exitCode

Basically, this script calls the following REST API endpoint:

http://RMSERVER/account/releaseManagementService/_apis/releaseManagement/OrchestratorService/InitiateRelease?releaseTemplateName=RELEASEDEFINITION&deploymentPropertyBag=PROPERTYBAG&api-version=3.0

This REST endpoint returns a release id, which we can use to read the status of the release. The script loops until the status is not “In Progress” and the quits and returns an exit code depending on how the release went.

The parameters of the above endpoint are:

RMSERVER
The URL including port to the release management server, typically somthing like contoso.com:1000
RELEASEDEFINITION
The name of the release template that we want to trigger.
PROPERTYBAG
Now this one is a bit special. This is an array of key/value which is not yet documented but it typically looks like this:

{ 
    "Component1:Build" : "Component1Build_20140814.1",
    "Component2:Build" : "Component2Build_20140815.1",                            
    "ReleaseName" : "$releaseName"
}

The first two lines references two different components in RM. Here I have just called them Component1 and Component2. The :Build is a keyword and must be there. The value part is the build number, which RM will append to the UNC path that we defined earlier. Note that we can use different build numbers per component here if we want to.

The last line is a predefined keyword that allows us to specify the name of the release, something that we actually can’t do using the RM client or the standard RM TFS Build release template so this is a nice feature.

So, to execute this script in TeamCity, add the above script to repository and then add a PowerShell build step at the end of your build in TeamCity that looks something like this:

The Script Arguments property is the tricky part, since we have to escape the quotes for the propertyBag parameter, and we will also use the %env:BUILD_NUMBER% variable from TeamCity for the build number. Here is the full string as an example:

-rmServer SERVER:1000 -rmUser USER -rmPassword PASSWORD -rmDomain DOMAIN -releaseDefinition VSGallery -deploymentPropertyBag “{“VSGallery Web Application:Build” : “%env.BUILD_NUMBER%”,”ReleaseName” : “%env.BUILD_NUMBER%”}”

Running this build in TeamCity will now trigger a release in Release Management: