Deploying ARM Templates using Visual Studio Team Services

If you are running your applications in Azure, and in particular on PaaS, you need to take a look ARM templates as a way to manage your environments. ARM templates let’s you define and deploy your entire environment using JSON files that you store together with the rest of your source code. The deployment of ARM templates are idempotent, meaning that you can run them many times and it will always produce the same result.

Image result for azure ARM templates

In this post, I will how you how to deploy ARM templates together with your application using Visual Studio Team Services. As you will see, I will not use the out of the box task for doing this, since it has some limitations. Instead we will use a PowerShell script to eexecute the deployment of an ARM template.

 

The overall steps are:

  • Defining our ARM template for our environment.
  • Tokenize the ARM template parameters file
  • Create a PowerShell script that deploys the ARM template
  • Deploy everything from a VSTS release definition.

Let’s get started with the ARM template.

ARM Template

In this case, I will deploy an ARM template consisting of a Azure web app, a SQL Server + database and a Redis Cache. The web app and sql resources are easy to deploy, since we can supply all the input from my release definition.
With the Redis cache however, Azure Resource Manager will create some information (such as the primarykey) as part of the deployment, which means we need to read this information from the output of the ARM template deployment.

Here is the outline of our ARM template:

image

 

Note the outputs section that is selected above, here we define what output we want to capture once the reource group has been deployed. In this case, I have defined three output variables:

  • redis_host
    The fully qualified edish host name
  • redis_port
    The secure port that will be used to communicate with the cache
  • redis_primatykey
    The access key that we will use to authenticate

Since our web application will communicate with the Redis cache, we need to fetch this information from the ARM template deployment and store them in our web.cofig file. You will see later on how this can be done.
 

Learn more about authoring ARM templates here: https://docs.microsoft.com/en-us/azure/azure-resource-manager/resource-group-authoring-templates

 

ARM Template Tokenization

When deploying our template in different environments (dev, test, prod…) we need to supply the information specific to those environment. In VSTS Release Management, the information is stored using environment variables.
A common solution is to tokenize the files that is needed for deployment and then replace these tokens with the corresponding environment variable.

To do this, we add a separate parameters file for the template that contains all the parameters but all the values are replaces with tokens:


{
    “$schema”: “
http://schema.management.azure.com/schemas/2015-01-01/deploymentParameters.json#”,
    “contentVersion”: “1.0.0.0”,
  “parameters”: {
    “hostingPlanName”: {
      “value”: “__HOSTINGPLANNAME__”
    },
    “administratorLogin”: {
      “value”: “__ADMINISTRATORLOGIN__”
    },
    “administratorLoginPassword”: {
      “value”: “__ADMINISTRATORLOGINPASSWORD__”
    },
    “databaseName”: {
      “value”: “__DATABASENAME__”
    },
    “webSiteName”: {
      “value”: “__WEBAPPNAME__”
    },
    “sqlServerName”: {
      “value”: “__SQLSERVERNAME__”
    },
    “dictionaryName”: {
      “value”: “__DATABASENAMEDICTIONARY__”
    },
    “extranetName”: {
      “value”: “__DATABASENAMEEXTRANET__”
    },
    “instanceCacheName”: {
      “value”: “__INSTANCECACHENAME__”
    }

  }
}


We wil then replace these tokens just before the template is deployed.

PowerShell script

There is an existing task for creating and updating ARM templates, called Azure Resource Group Deployment. This task let’s us point to an existing ARM template and the corresponding parameter file.

Here is an example how how this task is typically used:

 

image

 

The problem with this task is that it has very limited support for output parameters. As you can see in the image above, you can map a variable to the output called Resource Group. Unfortunately there is an assumption that the resource group that you are creating contains virtual machines. If you execute this task with an ARM template containing for example an Azure Web App you will get the following error when trying to map the output to a variable:

 

2017-01-23T09:09:49.8436157Z ##[error]The ‘Get-AzureVM’ command was found in the module ‘Azure’, but the module could not be loaded. For more information, run ‘Import-Module Azure’.

So, to be able to read our output values we need to use PowerShell instead, which is arguably a better choice anyway since it allows you to run and test the deployment locally,  saving you a lot of time.

When we create an Azure Resource Group project in Visual Studio, we get a PowerShell script that we can use as a starting point.

 

image

 

Most part of this script handles the case where we need to upload artifacts as part of the resource group deployment. In this case we don’t need this, we deploy all our artifacts from RM after the resource group has been deployed.

Here is our PowerShell script that we will use to deploy the template:


#Requires -Version 3.0
#Requires -Module AzureRM.Resources
#Requires -Module Azure.Storage

Param(
    [string] [Parameter(Mandatory=$true)] $ResourceGroupLocation,
    [string] [Parameter(Mandatory=$true)] $ResourceGroupName,
    [string] [Parameter(Mandatory=$true)] $TemplateFile,
    [string] [Parameter(Mandatory=$true)] $TemplateParametersFile
)

Import-Module Azure -ErrorAction SilentlyContinue

try {
    [Microsoft.Azure.Common.Authentication.AzureSession]::ClientFactory.AddUserAgent(“VSAzureTools-$UI$($host.name)”.replace(” “,”_”), “2.9”)
} catch { }

Set-StrictMode -Version 3

$TemplateFile = [System.IO.Path]::GetFullPath([System.IO.Path]::Combine($PSScriptRoot, $TemplateFile))
$TemplateParametersFile = [System.IO.Path]::GetFullPath([System.IO.Path]::Combine($PSScriptRoot, $TemplateParametersFile))

# Create or update the resource group using the specified template file and template parameters file
New-AzureRmResourceGroup -Name $ResourceGroupName -Location $ResourceGroupLocation -Verbose -Force -ErrorAction Stop

$output = (New-AzureRmResourceGroupDeployment -Name ((Get-ChildItem $TemplateFile).BaseName + ‘-‘ + ((Get-Date).ToUniversalTime()).ToString(‘MMdd-HHmm’)) `
                                   -ResourceGroupName $ResourceGroupName `
                                   -TemplateFile $TemplateFile `
                                   -TemplateParameterFile $TemplateParametersFile -Force -Verbose)

Write-Output (“##vso[task.setvariable variable=REDISSERVER]” + $output.Outputs[‘redis_host’].Value)
Write-Output (“##vso[task.setvariable variable=REDISPORT]” + $output.Outputs[‘redis_port’].Value)
Write-Output (“##vso[task.setvariable variable=REDISPASSWORD;issecret=true]” + $output.Outputs[‘redis_primarykey’].Value)


The special part of this script is the last three lines. Here, we read the output variables that we defined in the ARM template and then we use one of the VSTS logging commands to map these into variables that we can use in our release definition.

The syntax of the SetVariable logging command is ##vso[task.setvariable variable=NAME]<VARIABLEVALUE>. 

 

Note: You can read more about these commands at https://github.com/Microsoft/vsts-tasks/blob/master/docs/authoring/commands.md

 

Release Definition

Finally we can put all of this together by creating a release definition that deploys the ARM template.


Note
: You will of course need to create a build definition that packages your scripts, ARM templates and deployment artifacts. I won’t show this here, but just reference the outputs from an existing build definition.

 

Here is what the release definition will look like:

image

 

Let’s walk through the steps:

  1. Replace tokens
    Here we replace the tokens in our parameters.json file that we definied earlier. There are several tasks in the marketplace for doing token replacement, I’m using the one from Guillaume Rouchon (https://github.com/qetza/vsts-replacetokens-task#readme)
  2. Deploy Azure environment
    Run the PowerShell scipt using the Azure PowerShell task. This task handles the connection to Azure, so we don’t have to think about that.

    image

    Here I reference the PowerShell script from the build output artifacts, and also I supply the necessary parameters to the PS script:

    Script Arguments
    -ResourceGroupLocation “$(resourceGroupLocation)” -ResourceGroupName $(resourceGroupName) -TemplateFile “$(System.DefaultWorkingDirectory)/SampleApp.CI/environment/templates/sampleapp.json” -TemplateParametersFile “$(System.DefaultWorkingDirectory)/SampleApp.CI/environment/templates/sampleapp.parameters.json”

  3. Replace tokens
    Now we need to update the tokens in our SetParameters file, that is used by web deploy. It is important that we run this task after running the deploy azure enviroment script, since we need the output variables from the resource group deployment. Remember, these variables are now available as environment variables, so they will be inserted in the same way as the variables that we have defined manually.

  4. Deploy Web app + Deploy SQL Database
    These steps just performs a simple deployment of an Azure Web App and a SQL dacpac deployment.

 

That’s it, happy deployment! Smile

New Swedish Meetup Group for Microsoft ALM and DevOps

We have decided that it is time to create a meetup group for people that are interested in the Microsoft ALM and DevOps story!

image

 

Together with Mathias Olausson and a few other people we have created a new Meetup group and announced the first meeting.


Our plan is to continue meeting every month or so to learn about and dicuss new concept and ideas in the area of ALRM and DevOps on the Microsoft stack. This is a wide area, which spans all roles in the development process,
so there will be something for everyone.

 

First meetup: Microsoft Team Services Agile Transformation Story + VS ALM Update

The first meeting is set to October 25th, where we will have Jose Rady Allende, a Program Manager on the Visual Studio Team Services tean, join us online to talk about the Microsoft Team Service Agile Transformation story.
We’ll also going to have a few lightning talks where we will talk about recent new additions to the TFS/VSTS platform

Meeting link:

http://www.meetup.com/swedish-ms-alm-devops/events/234449734/

 

There are already around 25 people that have signed up for it, so sign up before it gets full!

 

Heop to see you there!

 

Image result for donovan brown devops

Using Web Deploy in Visual Studio Team Services Release Management

This post does not really cover something new, but since I find myself explain this to people now and then, I thought that I’d write a quick post on the subject.

So, we want to create a web deploy package as part of our automated build, and then take this package and deploy it to multiple environments, where each environment can have different configuration settings, using VSTS Release Management. Since we only want to build our package once, we have to apply the environment specific settings at deployment time, which means we will use Web Deploy parameters.

 

Here are the overall steps needed:

  1. – Create a parameters.xml file in your web project
  2. – Create a publish profile for the web deploy package
  3. – Set up a VSTS build creates the web deploy package, and uploads the package to the server
  4. – Create a Release definition in VSTS that consumes the web deploy package
  5. – In each RM environment, replaces the tokens in the SetParameters file

Let’s run through these steps in detail:

Create a parameters.xml file

As you will see later on, a publish profile contains configurable settings for the web site name and any connection strings,that will end up in the *.SetParameters.xml file that is used when at deployment time. But in order for other configuration settings, like appSettings, to end up in this file, you need to define these settings. This is done by creating a file called parameters.xml in the root of your web application.

Tip: A fellow MVP, Richard Fennell,  has created a nifty Visual Studio extension that simplifies the process of creating the parameters.xml file. It will look at your web.config file and the create a parameters.xml file with all the settings that it finds.

image

 

In this case, I have three application settings in the web.config file, so I end up with this parameters.xml file. Note that I have set the defaultvalue attribute for all parameters to __TOKEN__. These are the configuration values that will end up in the MyApp.SetParameters.xml file, together with the web deployment package. We will replaced these values at deployment time, by a task in our release definition.

<parameters> <parameter name="IsDevelopment" description="Description for IsDevelopment" defaultvalue="__ISDEVELOPMENT__" tags=""> <parameterentry kind="XmlFile" scope="\\web.config$" match="/configuration/applicationSettings/MyApp.Properties.Settings/setting[@name='IsDevelopment']/value/text()" /> </parameter> <parameter name="WebApiBaseUrl" description="Description for WebApiBaseUrl" defaultvalue="__WEBAPIBASEURL__" tags=""> <parameterentry kind="XmlFile" scope="\\web.config$" match="/configuration/applicationSettings/MyApp.Properties.Settings/setting[@name='WebApiBaseUrl']/value/text()" /> </parameter> <parameter name="SearchFilterDelta" description="Description for SearchFilterDelta" defaultvalue="__SEARCHFILTERDELTA__" tags=""> <parameterentry kind="XmlFile" scope="\\web.config$" match="/configuration/applicationSettings/MyApp.Properties.Settings/setting[@name='SearchFilterDelta']/value/text()" /> </parameter> </parameters>

Creating a Publish Profile

Now, let’s create a publish profile that define how the web deployment package should be created. Right-click on the web application project and then select Publish. Then select the Custom option:

image

 

Since the publish profile will be used to create a web deployment package, I like to call it CreatePackage (but you are of course free to call it whatever you want)

image

On the Connection tab, select Web Deploy Package as the publish method, then give the generated package a name (including .zip).

As the Web Site name, we enter a tokenized value __WEBSITE__. This token will also end up in the MyApp.SetParameters.xml file.

image

Save the publish profile and commit and push your changes. Now it’s time to create a build definition.

 

Create a Build Definition that generates a web deploy package

I won’t go through all the details of creating a build definition in VSTS, but just focus on the relevant parts for this blog post.

To generate a web deploy package, we need to pass some magic MSBuild parameters as part of the Visual Studio build task. Since we have a publish profile that contains our settings, we need to refer to this file. We also want to specify where the resulting files should be placed.

Enter the following string in the MSBuild Arguments field:

/p:DeployOnBuild=true /p:PublishProfile=CreatePackage /p:PackageLocation=$(build.stagingDirectory)

image

 

DeployOnBuild=true is required to trigger the web deployment publishing process, and the we use the PackageLocation property to specify that the output should be places in the staging directory of the build. This will make it easy to upload the artifacts at the end of the build, like so:

image

 

This will generate an artifact called drop in the build that contains all files needed to deploy this application using MSDeploy:

image

As you can see, we have all the generated web deploy files here. We will use three of them:

MyApp.zip – The web deploy package

MyApp.SetParameters.xml – The parameterization file that contains our tokenized parameters

MyApp.Deploy.cmd – A command file that simplifies running MSDeploy with the correct parameters

 

Creating a Release Definition

Finally, we will create a release definition that deploys this web deploy package to two different environments, let’s call them Test and Prod. In each environment we need to apply the correct configuration values. To do this, we have to replace the token variables in our MyApp.SetParameters.xml file.

There is no out of the box task to do this currently, but there are already several of them in the Visual Studio Marketplace. Here, I will use the Replace Tokens task from Guillaume Rochon, available at https://marketplace.visualstudio.com/items?itemName=qetza.replacetokens. Install it to your Visual Studio Team Services account, and then the task will be available in the build/release task catalog, in the Utility category:

 

image

 

Each environment in the release definition will just contain two tasks, the first one for the token replacement and the other one for deploying the web deploy package. To do this, we just run the MyApp.deploy.cmd file that was generated by the build. Since the parameters have already been set with the correct values, we can just run this without any extra arguments.

image

 

Also, we must specify the values for each variable in the environment. Right click on the environment and the add these variables:

image

 

Tip: Create the Test environment first with all variables and tasks. Once it’s done, use the Clone environment feature to create a Prod environment, and then just replace the configuration values

 

That’s it, now you can run the release and it will deploy your web application with the correct configuration to each environment.

 

image

 

 

 

 

TFS 2015 Update 2 RC1 Available – With VS Release Management vNext

Today Brian Harry announced that the first release of TFS 2015 Update 2 is available. It is an RC with a go-live license, which means that Microsoft will support you if you install it in production, and it will be a direct supported upgrade to RTM once it is released.

Read the full release notes for Update 2 RC1 here: https://www.visualstudio.com/en-us/news/tfs2015-update2-vs

 

One huge thing with the release is that Update 2 includes the new Release Management vNext feature that has up until now only been available in the service (although you can use that for on premise TFS and deployments). But now you don’t have to rely on VSTS for hosting the service, it is now part of TFS 2015 Update 2!

 

And of course, if you want to learn (a lot) more about hwo to implement continuous delivery using TFS 2015, with the new build and release management features, grab a copy of our latest book on the topic, Continuous Delivery with Visual Studio ALM 2015:

 

book

Deploy Azure Web Apps with Parameterization

I have blogged before about how to deploy an Azure Web App using the new build system in TFS 2015/Visual Studio Team Services. In addition to configure an Azure service endpoint, it is really only a matter of using the built-in Azure Web App Deployment task.

However, in many cases I have decided not to use this task myself since it has been lacking a key feature: Applying deployment parameters using the SetParameter.xml file.

As I have mentioned a gazillion times, building your binaries once and only once is a key principle when implementing continuous delivery. If you are using web deploy for deploying web applications (and you should), you must then use Web Deploy parameters to apply the correct configuration values at deployment time.

 

Under the hood the Azure Web App Deployment  task uses the Publish-AzureWebsiteProject cmdlet that uses web deploy to publish a web deploy package to Microsoft Azure. Up until recently, this cmdlet did not support web deploy parameters, the only thing that you could substitute was the connecting string, like so:

Publish-AzureWebsiteProject -Name site1 -Package package.zip -ConnectionString @{ DefaultConnection = "my connection string" }

However, it is actually possible to specify the path to the SetParameter.xml file that is generated together with the web deploy package. To do this, you can use the –SetParametersFile parameter, like so:

Publish-AzureWebsiteProject -Name Site1 -Package package.zip -SetParametersFile package.SetParameters.xml

When using the Azure Web App Deployment task, there is no separate parameter for this but you can use the Additional Arguments parameter to pass this information in:

image

Note: In this case, I am using the Azure Web App Deployment task as part of a release definition in Visual Studio Release Management, but you can also use it in a regular build definition.

This will apply the parameter values defined in the QBox.Web.SetParameters.xml file when deploying the package to the Azure Web App.

If you are interested, here is the pull request that implemented support for SetParameters files: https://github.com/Azure/azure-powershell/pull/316