Azure Stack Hub, if your not familiar with it (which is unlikely since you’re reading this walk through), is an extension of the Azure cloud built for offering a selection of azure services and capabilities in an on-premise solution that lives in your datacenter. In other words its a piece of Microsoft’s cloud solution that you can own and operate for yourself.
We recently were tasked with installing/deploying App Services to a clients Azure Stack Hub. Azure App Services on Azure stack Hub is a what is known a Platform-as-a-Service (PaaS) offering. With it you can integrate your apps with on premise apps and automate business processes. Some of the key features of this are: Multiple languages and frameworks, DevOps optimization, and Visual Studio Integration.
We deployed App Services RP 2020.Q2 version on Azure Stack Hub version 2002 – but from my understanding this experience is going to be largely the same from 1910 to 2005.
I put together this walk through because it can be extremely confusing to deploy app services when following the current Microsoft documentation and most other guides or walkthroughs on the process are outdated by 3+ years over which time the process has changed dramatically.
If you follow Microsoft documentation on installing app services you will notice a large list of prerequisites, its good to make sure you go through each one these. We’ll walk through these together before running the actual app services installer.
NOTE: It will take several hours to setup all the prerequisites and several more hours to install and finalize the installation of app services. Depending on your availability we suggest simply accomplishing this in stages over a period of time. Feel free to bookmark our article as=
Resource Provider Prerequisites
If you recently had your Azure Stack Hub deployed, it was likely that as part of the setup process your Azure Stack Hub instance was registered with Azure. If not you’ll want to see “Register your Azure Stack hub instance with Azure”
Download helper scripts and Installer
There are two main things you’ll want to download to begin:
- Deployment Helper Scripts for App Service on Azure Stack Hub
- Installer for App Service Azure Stack Hub
Extract the contents of the helper scripts zip file. You should see a handful of powershell scripts listed, including Create-AADIdenitytApp.ps1 and Get-AzureStackRootCert.ps1.
Important: When moving/copying this script from server to server, move/copy the entire root folder with all the scripts and subfolders with it, instead of just an individual script. These scripts have dependencies on files outside of the script itself.
Generate Certificates
Certificates?! I hear you loudly exclaim. Yes, certificates! In addition to the army of certificates you had to purchase during the original setup of Azure Stack Hub, another round of certificates must be purchased for App Services. Keep in mind Azure Stack Hub is a hybrid cloud solution so it makes sense that all these endpoints must be protected by a certificate from a valid public certificate authority.
There are 4 categories of certificates you’ll need to generate a request for and purchase from a valid certificate authority:
- Default Domain Certificates
- API Certificate
- Publishing Certificate
- Identity Certificate
1. For generating the App Services CSRs you’ll want to install the Microsoft.AzureStack.ReadinessChecker PowerShell module. Open PowerShell as an administrator on any computer you want to use to create the certificate requests. You’ll want to remember the computer this is done from because you will need to us this same computer to complete the certificates and retrieve the private keys later on. Run the following install module command to install the Microsoft.AzureStack.ReadinessChecker module.
Install-Module Microsoft.AzureStack.ReadinessChecker
2. In the same PowerShell session now create a subject variable for the cert. Example:
$subject = "C=US,ST=Washington,L=Redmond,O=Microsoft,OU=Azure Stack Hub"
3. Then create a variable for folder directory that already exists where the CSRs will be generated. Example:
$outputDirectory = "$ENV:USERPROFILE\Documents\AzureStackCSR"
4. Create a variable for your Azure stack region name and the external FQDN of your Azure Stack Hub. Example:
$regionName = 'phx'
$externalFQDN = 'azurestack.contoso.com'
5. Plug your variables into a request for new Hub App Services CSRs. Example:
New-AzsHubAppServicesCertificateSigningRequest -RegionName $regionName
-FQDN $externalFQDN -subject $subject -OutputRequestPath $OutputDirectory
This should output a list of Certificate Signing Request files (CSRs) to the output directory you defined in step 3.
Multidomain Wildcard certificate *IMPORTANT*: The wildcard_sso_appservice CSR is for a multidomain wilcard certificate. In other words its for a certificate that contains multiple wild card domains – in this case 3: *.appservice.., *.scm.appservice.., and *.sso.appservice… Not all Certificate Authorities offer a multidomain wildcard certificate. Unfortunately there is no way around this as the setup takes only a single pfx entry (src: default domain certificate)
We were concerned about needing to purchase a multidomain wildcard certificate as the rest of our certificates were provisioned by GoDaddy and they did not offer a multidomain wildcard option. Luckily we discovered that there was no technical reason why we couldn’t use a multidomain wildcard certificate from a different provider than the rest of our certs as long as that certificate authority is a member of the Microsoft Trust Root Certificate Program.
Important: Remember to keep track of which computer you generated the CSRs on. You will need to later import the returned certificate files into this same computer in order to get its private key and export the required .pfx files with certificate key.
When all is complete you should have 4 certificates in the form of .pfx files.
Validate Certificates
To validate the certificates add a ‘Certificates’ folder to your C: drive with a sub folder called AppServices and create 4 subfolders under that for each of the certificates: API, DefaultDomain, Identity, Publishing.
Inside each of the subfolders, place the appropriate .pfx certificate file
Import the Microsoft.AzureStack.ReadinessChecker module (assuming it is installed):
Import-Module Microsoft.AzureStack.ReadinessChecker
Use the Invoke-AzsHubAppServicesCertificateValidation cmdlet to run a validation check. Enter the PFX file password, your stack region name and you azure stack FQDN. For instance if your azure stack instance is phx.azurestack.domain.com, then your region is phx, and your FQDN is azurestack.domain.com (we’ve also seen some clients use their top level domain for the FQDN, in that case, the instance would be like astack.domain.com where their region is astack and their FQDN is domain.com)
Invoke-AzsHubAppServicesCertificateValidation -CertificatePath C:\Certificates\AppServices
-pfxPassword $pfxPassword -RegionName east -FQDN azurestack.contoso.com
You should then see each .pfx file go through a validation process.
If all you see are green “OK”s for each certificate, then you are good to go. Otherwise there maybe an issue with the certificate you have.
Azure App Services Prerequisite Infrastructure
Azure App Services on the Azure Stack Hub requires some underlying infrastructure to be deployed including a domain controller, SQL server, and file share.
But before you get concerned with adding a myriad of new VMs in your azure stack environment, take note that this infrastructure is ‘behind the scenes’ and only viewable from the azure stack adminportal, not the standard portal.
Fortunately, to ease and quicken this infrastructure build, a quick start template was released : Quickstart template for Highly Available file Server and SQL Server
From the GitHub containing the appservice-fileserver-sqlserver-ha template files, locate the file called azuredeploy.json. It is with this json file that we will create a new template in azure stack.
Deploy Template
IMPORTANT: Before deploying this template, make sure you have downloaded “Windows Server 2016 Datacenter – Server Core” either pay as you go or BYOL and the “SQL IaaS Extension” from the Azure Stack marketplace. Both are not readily mentioned in the prerequisite documentation but missing either will cause the template deployment to fail.
From the azure stack hub admin portal (should be similar to format adminportal.region.azurestack.domain.com) select “Create a resource”.
On the ‘create a resource’ blade type in “template” into the empty search bar.
select ‘Template deployment’ and on the following blade select ‘Create’.
On the ‘Deploy Solution Template’ blade select the ‘Edit template’ button
In the ‘Edit template’ blade you will find an option to ‘Load file’, which would allow you to simply upload the .json file from the template, however since this is a quickstart template we have the ability to pull this template directly from a drop down by first selecting ‘Quickstart template‘.
Select the ‘appservice-fileserver-sqlserver-ha’ quickstart template.
You should see the description ‘Creates VNET, 2 node SQL cluster, 2 node FileServer cluster and 2 domain controllers’.
Select OK to continue.
This should load in the template and its parameters. Select Save.
Now if you check the parameters blade you will see prefilled in parameters. Leave the defaults or customize as desired. We made a few customizations but left most settings as was default. Fill in the remaining blank parameters which should mainly be passwords for administration, fileshare and sql server. Select “OK” when you’ve completed filling out the parameters blade.
For resource group we recommend creating a brand new group such AppServices_RG, this way if anything is wrong with the deployment you can simply delete the resource group and redeploy.
Resource group location should contain the region of your azure stack.
Select ‘Create’ from the bottom of the Deploy Solution Template blade.
The deployment may take roughly 2 hours.
If you run into any failures, they are likely because you did not download the “Windows Server 2016 Datacenter – Server Core” either pay as you go or BYOL and the “SQL IaaS Extension” from the Azure Stack marketplace. Neither are readily mentioned in existing documentation but halted our progress none the less.
When it is all completed you should see all the deployments listed as succeeded.
You should see new virtual machines in the adminportal of the azure stack. 
The template creates:
- A resource group
- A /16 Virtual Network and 8 Subnets (ADSubnet, SQLSubnet, FileServerSubnet, ControllerSubnet, ManagementSubnet, FrontendSubnet, PublisherSubnet, WorkerSubnet)
- Network Security Groups for each subnet
- 2 Windows Server 2016 Datacenter Server Core domain controllers
- 2 Windows file servers setup for high availability
- 2 SQL servers setup for High availability
Retrieve the Azure Resource Manager Root Certificate for Azure Stack Hub
Microsoft documentations says to do this by running an “elevated PowerShell session on a computer that can reach the privileged endpoint on the Azure Stack Hub Integrated System” however it may be confusing as to what actually has access to the privileged endpoint. After some trial and end error we discovered that this can be successfully run from the Hardware Lifecycle Host (HLH). The HLH is used for solution deployment, monitoring and management. This particular setup is an HPE solution and this may be different for other vendors of Azure Stack hub.
1. Connect, via remote desktop to the HLH IP address
2. Copy the Deployment Helper Scripts for App Service on Azure Stack Hub to the HLH. Make sure to have copied the entire extracted folder structure and not just a single script.
3. Run the Get-AzureStackRootCert.ps1 script from the folder containing the extracted helper scripts. The script creates a root certificate in the same folder as the script that App Service needs for creating certificates.
When running the script using the CloudAdmin (AzureStack\CloudAdmin) account for your Azure Stack Hub.
Create Azure AD Identity Application
Azure App service uses an Identity application (service principal) to support virtual machine scale set integration on worker tiers and SSO for the Azure functions portal and advanced developer tools.
To create this you will need to run the Create-AADIdentityApp.ps1 script from the folder of helper scripts you downloaded earlier.
As with the Root Certificate, the Create-AADIdentityApp.ps1 script needs to be run from a computer that can reach the privileged endpoint on the Azure Stack Hub, again we found this to be the Hardware Lifecycle Host (HLH).
1. Connect, via remote desktop to the HLH IP address
2. If you have not already, copy the Deployment Helper Scripts for App Service on Azure Stack Hub to the HLH. Make sure to have copied the entire extracted folder structure and not just a single script. Also copy the sso.appservice… certificate pfx file you generated earlier with the rest of the app services certificates.
3. Open a PowerShell instance as azurestack\AzureStackAdmin and run the Create-AADIdentityApp.ps1 script.
4. When prompted, enter the Azure AD tenant ID. This could be 1 of two possible format – domain.onmicrosoft.com, or the primary vanity domain you are using in Azure Active directory (domain.com)
5. You’ll be asked for the AdminArmEndpoint, this should resemble the format adminmanagement.<region>.<azurestack>.domain.com
6. TenantArmEndpoint will be similar minus the ‘admin’, typically in the format management.<region>.<azurestack>.domain.com
7. When asked for the CertificateFilePath enter the full location of the sso.appservice pfx file you copied to the HLH earlier. When asked for the Certificate password, it should be the password you set when you extracted the certificates to .pfx files.
8. You may be warned about running another PowerShell script, again make sure you have copied the entire extracted file structure of helper scripts to the HLH instead of just the individual ps1 file. Run the script it calls on.
9. Once completed you should be presented with an application ID, save this.
10. Go to the Azure global portal (portal.azure.com) and to Azure Active Directory and select “App registrations” from the blade. make sure ‘all applications’ is selected and search for the application ID that was given from the previous step and select that application from the search results.
11. On the application blade, select the ‘API permissions’ and select the “Grant admin consent for (application name)”
You’ve now successfully setup the Azure AD Identity application for app services on azure stack.
That should complete all the prerequisite steps for Azure App Services on Azure Stack Hub, however you may also want to check that you have the latest version of Windows Server 2016 Datacenter VM image and the Custom Script Extension v1.9.1 or greater downloaded from the Azure marketplace on the azure stack hub.
Run the App Service resource provider installer
After what may seem like an endless adventure of prerequisite work to get here, it is finally time to deploy the actual Azure App Services by running the App Service resource provider installer – the .exe file you should have downloaded earlier (if you didn’t it can be downloaded here: App Service on Azure Stack Hub installer).
So what does the installer even do?
The installer will run the following tasks:
- Registers the required resource providers in the Default Provider Subscription
- Grants contributor access to the App Service Identity application
- Create Resource Group and Virtual network (if necessary)
- Create Storage accounts and containers for App Service installation artifacts, usage service, and resource hydration
- Download App Service artifacts and upload them to the App Service storage account
- Deploy the App Service
- Register the usage service
- Create DNS Entries for App Service
- Register the App Service admin and tenant resource providers
- Register Gallery Items – Web, API, Function App, App Service Plan, WordPress,
- DNN, Orchard, and Django applications
Where should we run this from?
Great question! Its the same questions we asked and after a few trial and error attempts we decided to spin up a new virtual machine (in our case we called it APP-MGMT) from the stack admin portal in the same virtual network as the VMs deployed in the prerequisite steps. Basically creating a jumbox into this backend environment.
Doing this made sure the installer we as able to talk to the local active directory created for app services as well as the rest of the required endpoints (SQL & File server).
1. Remote onto the new management VM you created, it will need a ‘public IP’ address assigned to it to make it reachable via remote desktop.
2. Copy the appservice.exe to that new management VM while logged in as a local admin.
3. Run the appservice.exe file and select ‘Deploy App Service or upgrade to the latest version’
4. Read and agree to the terms and select ‘Next >’
5. Enter the hub Admin ARM Endpoint (should start with adminmanagement) and the Hub Tenant ARM endpoint (should start with management) and the Azure Directory Tenant Name, this is either in the formation of domain.onmicrosoft.com or domain.com where domain.com is your primary domain in azure active directory. Then select ‘Next >’
6. On the next page you must select a connection method for the Azure App Service cloud configuration. We selected to use ‘Credential‘, which allows you to us the Azure AD admin account and password you used to setup the azure stack hub initially. Alternatively you can setup a service principal. Select and fill out the desired method and select ‘Next >’
7. After connecting you can then select whether to create a VNet with default settings or use an existing VNet and Subnets, since we ran the quickstart template, those pieces have already been created – select ‘Use existing VNet and Subnets’, select the Resource Group that you created when deploying the quickstart template and select the virtual network and the subnets that correspond with each listed subnet. Select ‘Next >’ once completed.
8. On the next installation page enter the path for your file share, this should be found on the output for the quickstart template deployment.
Find template output
You can find information like the File Share UNC Path by going to the azure stack hub admin portal and opening up the resource group you deployed the quickstart template to and on the deployment blade select the template deployment.
You should see the filesharepath, owner, user, and sql server and user info listed. Password will be what you entered in the parameters during the deployment of the template.
9. On the next App Service installer page enter the Application ID from when you configured the Azure AD Identity Application during the prerequisite setup.
Then enter the path to the Identity application certificate pfx file. This should be the sso_appservices certificate. The password should be whatever you set it to when you extracted the .pfx file.
Then enter the Azure Resource Manager root certificate file that you extracted earlier in the prerequisite setup while on the HLH server.
Select ‘Next >’ after completing
10. On the next page enter in the App Service default SSL (this should be the multidomain wildcard certificate), Api SSL (the api_service certificate), and Publisher certificate (the ftp_appservice certificate), their password will be whatever you set them to when you exported the .pfx files.
11. You should be able to use the IP address or name of the sql server, the App Service installer tried to test connectivity to SQL before proceeding but if your using an existing virtual network, which in this case WE ARE, then documentation says the test might fail. In which case we receive a warning and prompt to continue. Simply make sure you have the correct name or IP (as well as the credentials) and select ‘Next >’
12. The next page allows you to specify number of role instances and SKU options. We suggest following at least Microsoft’s minimum recommended number of instances and compute SKU. For further details refence ‘Capacity planning for App Service server roles in Azure Stack Hub‘ otherwise refer to image below.
Additional worker configuration can be done post deployment from the stack admin portal. When you have set the desired number instances and SKUs, select ‘Next >’
13. On this install page, in the ‘Select Platform Image’ choose your Windows Server 2016 VM image from the images available. (keep in mind that core is not supported here)
14. On the next page provide the worker role VM admin and password and Other Role VM Admin and password, then select ‘Next >’
15. Review the summary page, and select the box ‘Select and click next start the deployment’, then select ‘Next >”
16. Track progress of the installation it can take around 2 hours or more depending on the deployment.
Post Deployment Tasks
So if you’re like us, you might be tempted to think “the install is done – app services must be ready to go!” But there are 2 very important things to do before your app services is ready.
1. Add the appservice_hosting and appservice_metering databases to availability group and synchronize
If you do not complete this task, you may run into issues accessing the AppService blade in the Azure Stack hub admin portal (which can be reached via All Services > Administration > App Service.
Why? Because of the SQL Always setup. A SQL Always on listener controls access the SQL databases and when the listener connects to the 2nd node its unable to find the databases since they are not being synced frome node 1, this causes the the AppSevices extension to fail to load.
To complete this task, log into the management server you created previously that is in the same vnet as the rest of the app services infrastructure.
From the management server remote to the SQL server using its private IP address and the domain administrator account which should have access to the databases.
On the sql instance, expand the folder structure and then expand the ‘Always On High Availability’ folder. 
You should see an availability group named something like ‘alwayson-ag (Primary)‘, rich click the availability group and select ‘Properties’. Note that there are no databases listed.
To add databases, select the ‘Add’ at the bottom of the Availability Databases window, or you can right click the availability group and run the wizard to add the databases. Both should achieve the same results.
IMPORTANT: However before being able to add the databases, full backups must be taken of them.
You will need to choose an account to use to connect the SQL databases for replication.
Once successfully added you should see the appservice_hosting and appservice_metering databases listed under ‘Availability Databases’
2. Add outbound security rule for SMB traffic
Since we deployed to an existing virtual network and using an internal IP address to connect to file server we have to add an outbound security rule. If you don’t, it will prevent SMB traffic between the worker subnet and the file server, preventing the work roles from finishing to install and the app service from completing.
Find the WorkersNsg Network Security group in the admin portal. Create the following outbound rule:
- Source: Any
- Source port range: *
- Destination: IP addresses
- Destination IP address range: Range of IPs for your file server
- Destination port range: 445
- Protocol: TCP
- Action: Allow
- Priority: 700
- Name: Outbound_Allow_SMB445
Validate App Service on Azure Stack Hub
Open App Service in the Azure Stack Hub admin portal and verify the status says “All roles are ready”.
And that’s it! You’re done with installing Azure App Service on Azure Stack Hub! You can now add it to an offer and plan.
Let us know your experiences with installing Azure App Service and if you have any questions we’d be glad to help. Send your questions/comments to jeremy.wallace@safarimicro.com
Need more assistance with your Azure Stack Hub? Reach out to us today to get in touch with one of our engineers.