In some cases you want to grant someone access to your Azure environment, but you don’t want them to connect from everywhere or at any given moment. But if a user is given a role in Azure, he or she can access the environment from everywhere with the rights, configured in their assigned role.
To solve this, Managed Identities are an easy way to grant access to a user, but only on a certain virtual machine.
In this short article, I will show you how you can grant access this way.
A managed identity is an Azure Active Directory object that can be used to authenticate. It comes in 2 flavors:
In case you use this on a single resource, a system-assigned managed identity will do, but for using it on more than one resource, a user-assigned managed identity is needed.
To create a managed identity on a virtual machine, simply turn the status on:
Now, this virtual machine has a managed identity. But this doesn’t mean it has any access. You still need to assign an Azure role to this managed identity. This is very similar to assigning a role for a normal user.
First select to what you want to assign access to:
Then select the virtual machine.
Of course, you still need to select the proper Role and then you are ready to roll.
So now you have created a managed identity and you have granted access to Azure. But how does this work?
In PowerShell the only command, needed to login is:
1 | Add-AzAccount -Identity |
For the Azure CLI, this command is used to login:
1 | az login --identity |
After you are logged in, you can manage the Azure resources you have access to.
This is a quite simple solution to grant access from a virtual machine you have under control, without giving users access from elsewhere. But there are some caveats:
When you get more experienced in Kubernetes, you get quite handy in using aliases for kubectl and short names for resource types. But sometimes it can be nice to have a graphical insight in your environment.
So, I heard about a tool that gives you this, Octant.
One of the most important reasons for using a tool, other than the Kubernetes dashboard is security. The Kubernetes dashboard is not insecure, but when not implemented well, it can create a security breach.
Octant is not installed within Kubernetes, but it uses the same way of connecting to Kubernetes as kubectl. This means that you are limited to see only what you are allowed to see, but in a very secure way.
So, before you can use Octant, you need to be logged in to your Kubernetes cluster.
The installation of Octant is very easy. Personally, I use Chocolatey for the installation of Octant, but it is also possible to download an installer from the releases page in github.
For chocolatey:
1 | choco install octant --confirm |
In order to run Octant, you need to be logged in to the cluster using kubectl, so you should need a config file under ~/.kube.
Then you’re all set. Type octant in a command or PowerShell terminal and a browser window will pop-up with Octant.
When you click on one of the recource types, you hit the overview page of the resources of that type.
You can now click on one of the resources that will show you a summary.
Another nice view is the Resource Viewer, which shows you a more relational view of the resources.
The last view, available for the resources is the YAML view. This gives you the complete configuration of the resource.
The summary page depends on the resource type that is selected. For example, if you take a pod, you have information about the resources, volumes and events:
Resources and volumes
Events
Another very cool thing about Octant is that you can create a port-forward, directly from the web-ui. When created, a direct link is shown that opens a new tab. It’s not difficult to create a port-forward with kubectl, but this way it’s extremely easy.
Currently, port-forward is limited to pods. When you want to port-forward to a service, then you still need kubectl.
To create a port-forward from the pod summary, click START PORT FORWARD:
To end the port-forward, click STOP PORT FORWARD:
I’m a fan of this open source project. You have a very nice and detailed insight in your environment. You see what you need, nothing more, nothing less. Furthermore, when you are logged in to Kubernetes, which is most likely the case, then it all comes to installing and running one command to get started!
Please, feel free to give your comments or share your experience below.
]]>Getting started with Kubernetes can be quite hard. There are some very good books, like ‘Kubernetes in Action’ from Manning or ‘Kubernetes the hard way’ on Linux Academy. But you will certainly agree that learning works best when you get started with a nice project that is easy to understand or at least well documented.
The project I’m presenting here is a php page, served from 3 pods, spread over 3 nodes. An nginx ingress controller is used to enable ingress network traffic and cert-manager is used to create a certificate, signed by Let’s Encrypt. Here a high level overview:
This guide is valid for nearly every cloud provided Kubernetes cluster. I clearly state ‘cloud provided’ for a reason. You can host your own Kubernetes cluster in Minikube, or have it self-hosted on physical or virtual machines. But using a cloud provided cluster, some extra needed resources, like load balancers or storage are provisioned automatically. This is very convenient. For my blog I’m using an Azure Kubernetes Service (AKS) cluster. But using another cloud will work as well with minimal changes.
You will also need an Ingress Controller. There are a lot of Ingress Controllers available like nginx, traefik, … In this guide, we are using nginx.
These days, we also want to protect our websites with a certificate. Therefore, we will also need cert-manager to request a certificate from Let’s Encrypt using the ACME protocol.
I will show you the commands to install the nginx Ingress Controller and cert-manager in a minute.
We also have an extra requirement, the nfs-server-provisioner. This will be explained in more detail in the storage chapter.
The last requirement we have is Helm, a package manager for Kubernetes. We will install it first, since we need it to install the rest of the requirements.
I won’t go too deep into detail for the installation of a Kubernetes cluster in Azure (AKS). This is a well documented process, which can be found here:
The installation of Helm is also documented well. The installation guide can be found here:
You can install Helm on almost any operating system. Personally, I’m working with Windows 10, but I use the Windows Subsystem for Linux (WSL) with Ubuntu for managing AKS. I also like to recommend the new Windows Terminal, which supports tabs and multiple types of terminals (like PowerShell, PowerShell Core, CMD, Linux, …).
For the installation of the nginx Ingress Controller, we’ll use Helm that we have installed in the previous step:
1 | # If you don't have the nginx-stable repo installed |
The installation of the cert-manager, that will handle the request of the certificate, is also done using helm and is therefore very straight-forward:
1 | # If you don't have the cert-manager-stable repo installed |
Since we will deploy a 3 node setup, we will need to use some kind of shared storage. There are a few ways to present storage to your pods, but the best way to do so is to make use of Persistent Volumes. This is an abstraction of the storage, which is very well explained in ‘Kubernetes in Action’, but in a nutshell explained:
An administrator creates a Persistent Volume (PV) using a Storage Class (SC) that is suited for the back-end storage. In AKS, the default Storage Class creates standard HDD managed disks. When the Persistent Volume is created, the developer, who doesn’t care about back-end disks, just needs to create a Persistent Volume Claim. The Persistent Volume Claim reserves a part of the Persistent Volume.
Another very important concept is the Access Mode. This can be ReadWriteOnce (RWO) or ReadWriteMany (RWX). The first one can only be accessed by one pod at a time while the latter can be accessed by multiple pods at once. In this example, using 3 nodes, we will need an RWX access policy.
In Azure you can choose for Managed Disks or Azure Files. Managed Disks only support RWO, where Azure Files also support RWX. At the time of writing this article, using Azure Files is very slow, even with Premium disks and for test purposes. Therefore, we have to work around this.
The workaround, to present Managed Disks to multiple pods, is to create an nfs-server-provisioner. This provisioner creates an NFS Storage Class, supporting RWX and connects to an existing Storage Class (e.g. default). With this intermediate Storage Controller, RWX can be provided to the pods.
To install the nfs-server-provisioner, helm is also used. But this time, some extra parameters are used to configure the back-end storage for the NFS Storage Class.
1 | # When you don't have the stable repo installed |
Here’s the output of the storage classes with NFS when everything went well:
As you see, I don’t use kubectl, but k and I don’t write StorageClass, but sc. The main reason for this is because I’m as lazy as an IT Pro should be. And, playing around with Kubernetes makes you type kubectl so many times that it becomes simply too long… Therefore, I created an alias for the kubectl command. When specifying Kubernetes resource types, a short name can be used in a lot of cases. The list with all resource types with abbreviations can be found here.
I didn’t specify any namespace in the above commands, but Kubernetes uses namespaces to separate environments.
You can specify a namespace with both helm and kubectl by adding the –namespace
1 | # Create a new namespace |
Important
Some resource types, like nodes, are not namespaced. This can also be found on the resource types page, linked above.
Now, with all prerequisites in place, we can get started with the actual deployment. It consists of the following components:
The Replication controller contains the definition of the pods with their containers. There is also a number of replicas specified. When deployed, the replication controller schedules a specified number of pods.
A volume is also created, pointing to the Persistent Volume Claim, and mounted to “/var/www/html”.
1 | apiVersion: v1 |
Don’t deploy this deployment file yet as this is only a part of the deployment.
There is no connection possible from outside, directly to a pod. To enable this, a service is needed. The service object creates an entry point, like in this case a cluster IP. The service will forward traffic to all the pods where the label matches the selector of the service.
1 | apiVersion: v1 |
The Persistent Volume Claim uses the NFS Storage Class. It is also necessary to define the ReadWriteMany (RWX) access mode to enable access from multiple pods at the same time. We’ll claim 2Gi (2 Gibibyte).
1 | apiVersion: v1 |
The Ingress object handles external access to the service object. It points to the previously installed nginx Ingress Controller. In the specs, the host specification in the ingress controller, the hostname is specified to which the controller responds. If traffic hits the ingress controller, but pointing to a different host, then the traffic will be dropped. The path is the location within the url, after the hostname.
In the TLS section, the hostname is the name on the certificate. It obviously needs to match the host in the previous section. The secret contains the certificate and private key that is installed in the Ingress Controller.
There are also some annotations that create some settings:
1 | apiVersion: extensions/v1beta1 |
1 | # Find the IP address of the ingress |
Important
Don’t forget to register the hostame in public DNS. It needs to be resolvable by your machine. This could be fixed with your hosts file. But Let’s Encrypt needs to be able to access the validation URL as well. And therefore, hosts file is not enough and public DNS should be registered.
A Cluster Issuer is used to obtain certificates from a CA, like in this case Let’s Encrypt. The difference between an Issuer and a Cluster Issuer is that the latter can be referenced by objects in other namespaces.
The ACME server, specified in the definition, is obtained from Let’s Encrypt. A valid email address within the domain needs to be specified.
Since the cert-manager uses http01 as solver, it will create a url within the domain (e.g. http://kubernetes.groovesoundz.be/.well-known/acme-challenge/qlIfs-sflkjsdgflskdfFSDQ34Fffz3RZFDSS). This will prove the ownership of the domain. An ingress controller will point this path on the hostname to a pod from the cert-manager, which will approve the request of the certificate.
1 | apiVersion: cert-manager.io/v1alpha2 |
The deployment file consists of all the parts above with a line with 3 hyphens ‘—‘ (without the quotes obviously) in between.
Website.yaml1
2
3
4
5
6
7
8
9<ReplicaController>
<Service>
<PersistentVolumeClaim>
<Ingress>
<ClusterIssuer>
This deployment can also be found on my github: https://github.com/christofvg/kubernetes-3-node-php-site.
To deploy the actual deployment file, a deployment needs to be created in Kubernetes:
1 | # Create a deployment, based on the deployment file |
The deployment creates the pods with shared storage. But this shared storage is empty of course. The point of this article is not to create a fancy website, but to show the workings of Kubernetes. Therefore, the content can consist of only one line that displays the pod’s name (hostname).
Here’s how to add the content:
1 | # find out the names of the pods |
From now on, the hostname (the name of the pod) will be shown when the pod is accessed through the ingress and service. The load will be balanced over the pods, so when you refresh, it changes. It might stay the same a few times. This is because of the way a browser handles connections. When you use curl to connect to the URL, then it changes every time since every time a new connection is started.
We have created a high available web server environment, complete with load balancing and certificate. I also tried to do this in a way, explaining things I found hard at first.
Please use the comments below to share your experiences.
]]>In my previous post “PowerShell GUI with externally managed data“, we created a PowerShell GUI script that uses externally managed data. But wouldn’t it be great if the application was just an executable, which could be easily used by end-users, and that could be deployed using Mobile Device Management? Of course! In this blog post, I will guide you through the process of updating your pipeline to automatically convert the PowerShell script into an executable.
The code of the GUI project, including the conversion to an executable, can be found on the github repository.
For the conversion from a PowerShell script to an executable, I use the tool PS2EXE. This tool was originally created by Ingo Karstein, but is currently maintained by Markus Scholtes.
Next to a conversion to an executable, the script is also capable of setting meta data in the executable, like a description, company, …
This tool is added to the repository and will be called by the build script.
The build step, that will convert the PowerShell script to an executable, needs to run a PowerShell script as described here:
1 | .\PS2EXE\ps2exe.ps1 -inputFile '.\Bin\EndpointManager.ps1' ` |
The script is very self-explanatory. We provide the script with an input file, which is the PowerShell GUI script. The output file is the executable that will be created.
The icon file parameter is used to change the default PowerShell logo of the application to a custom logo. Then, parameters follow that set the meta data of the file.
I recommend using semantic versioning with the build part set to the build number. This creates a traceable link between the application and the build version, making it easier to troubleshoot when needed.
Some extra parameters are passed to the script:
With these parameters set, an executable is created that is not different to the end-user than any other application. Double click the icon and a window pops up without any PowerShell window.
So we added the PS2EXE files to the repository and created a script to convert the PowerShell script to an executable. Now, it is time to run the script from the build pipeline.
To do so, add a PowerShell script task to the build pipeline and call the script you created:
With the meta data set, the properties look like this when showing the properties of the executable:
The icon is also set to the custom icon:
Adding a step to convert the PowerShell script to an executable automatically is very easy and not very much more than a oneliner.
Feel free to leave any comments below, or send me a message over twitter.
]]>A while ago, I got a question from a customer to create a PowerShell GUI script to be used by their end-users. The customer is a global company, with multiple offices in various countries. This script had following requirements:
The suggestions I made were also accepted:
In this blog post I want to show you how I created an PowerShell script with a GUI with one release pipeline, and another pipeline with the data, used in the script.
I created the application using the following tools:
All code can be found on my github repository. Unfortunately, GitHub doesn’t support projects containing multiple repositories. Therefore, I created a folder structure to separate all code. But it will give you a good view about how I created this code and these pipelines.
An important requirement was to filter the data, based on a selection of the country and the location of the end-user. To do so, I created a data array with an object per country/location. This object contains an array for the printers, shares and phone numbers.
1 | { |
On top of the JSON file, I added a version. This version will be showed in the application later on. When the end-user calls the Service Desk for an issue with the app, it can be easily determined if the end-user has the latest version of the data.
Important
The version is 1.0.0 in the file. No worries! This is updated in the automated build, changing the version to the correct build version.
The build process has 2 tasks:
1 | # Convert the JSON file to a PS object |
The only step in the release process uploads the file to a blob storage in an Azure Storage Account.
Creating the PowerShell GUI itself is out of scope of this blog post. If you like to know more about creating PowerShell GUI scripts, I recommend reading the “Learning GUI toolmaking series” by Stephen Owen.
Most articles describing PowerShell GUI scripts put the XAML code directly in the code of the script. Not only does it make your code longer, there is not a nice separation between the PowerShell code and the GUI code. When using Get-Content of the XAML file, code remains cleaner and the Visual Studio solution can exist in your repository as a separate directory.
1 | $inputxml = Get-Content '..\Endpoint Manager\MainWindow.xaml' -Raw |
Since we uploaded the JSON file to an Azure Storage Account, we need to download it to be used in the script. You can store it wherever you want. I used AppData, which avoids issues with rights for the end-user and it is a logical location for the data.
1 | Invoke-WebRequest -Uri "<URL to the storage account - with key if needed>" ` |
Important
I used ‘SilentlyContinue’ as error action. The reason for this is because this way, the application will use the latest available version of the data if no internet connection is available without crashing.
The support information for the end-user contains all information that the Service Desk might need to support the end-user. The most important information in this context is the version information of the data file.
1 | # Get the version from the JSON file |
When the country and location are selected, all filtered data can be displayed.
1 | $WPFcmbLocation.add_SelectionChanged({ |
This is the result I got…
System information:
Printers:
Shares:
It was actually not that hard to create the GUI application with a pipeline for the application itself and for the JSON data file. Making changes is very easy with the support of Azure DevOps. Data can be entered now by people who manage data without any programming skills.
]]>We now have the foundation of our Virtual Datacenter in place. We created a central hub, meant to accommodate centralized services like firewalls, domain controllers, file servers, … . Spoke networks are created for specific workloads that need to be separated from other workloads for security or governance purposes. Another network is created that will be connected using a site-to-site ipsec tunnel to simulate an on-premises network. With all these networks in place, we are ready to implement the centralized firewalls that will inspect and control all east-west traffic (between the spokes and the on-premises network) and north-south traffic (between the internal networks and the internet).
The ARM templates for the deployment are available on my GitHub page so I won’t put the files here. But we will go deeper into certain parts of the ARM templates in this article where needed.
When you want to deploy the firewalls as described in this article, I assume that you deployed the networks as described in the previous article. If you did not deploy te networks, the deployments will fail since the templates depend on these networks.
For this lab, we will deploy pfSense firewalls as NVAs (Network Virtual Appliances). These firewalls are very user friendly and are perfect to be used to learn networking through NVAs in Azure. In a production environment, I would recommend to use pfSense NVAs from the marketplace as they are supported by Netgate. We will deploy custom pfSense images, created in another blog series to enable a high level of automation (the marketplace pfSense images only have 1 nic and PowerShell intervention is needed to add more nics).
In these articles, I’ll show you how to create a pfSense image that can be used in Azure:
Since we will deploy more than one firewall, it is necessary to load balance the traffic between both firewalls. As we saw in the design decisions in Part 1, we are using Standard load balancers. This new type of load balancers have many advantages over Basic load balancers, but the main reason we will use Standard load balancers is because they support HA ports. With those HA ports, you can load balance all traffic instead of a set of specific ports which is very limited with Basic load balancers.
To load balance traffic on the trusted and on the untrusted side of the load balancer, we will use 2 separate load balancer instances. It is important that you use the same SKU for both load balancers (Standard in this case). SKUs may not be mixed between different resource types either. So, the public IP of the untrusted load balancer needs to be a Standard SKU public IP as well.
The firewalls will be placed in an availability set for high availability. Placing them in an availability set will make sure they are spread over different fault domains and update domains. Placing the firewalls in different fault domains means that they will be spread over hypervisors in different physical racks with different power, cooling and hardware. Update domains separate the virtual machines on different underlying hardware that won’t undergo maintenance or reboots at the same time.
For the deployment of the resources, we use ARM templates, describing the required resources. But before the templates can be deployed, Azure Resource Groups need to exist. In a later post, I will explain how to deploy the resources, including the resource groups.
Following resources are deployed with the ARM template under HUB firewalls:
Public IP Address: The public IP Address for the untrusted load balancer
Untrusted load balancer: The load balancer on the WAN side of the firewalls
Trusted load balancer: The load balancer on the LAN side of the firewalls
Availability set: The availability set for the load balancers
Untrusted nics: The network interfaces on the WAN side of the firewalls
Trusted nics: The network interfaces on the LAN side of the firewalls
Virtual machines: The firewall virtual machines
OS disks: The OS disks (from VHD image) that needs to be attached to the firewalls
The deployments of the nics, the virtual machines and the OS disks are using copy functions. This allows you to deploy multiple instances of the resources, based on a “number of instances” parameter.
Code snippet for the copy function:
1 | "copy": { |
The untrusted load balancer has 1 load balancing rule, TCP port 443 (https). This will spread the load on port 443 TCP on all back-end nodes.
1 | "loadBalancingRules": [ |
A health probe also checks the status of port 443 TCP. If it is reachable, the node is considered “online”. When it is not answering, then the node is considered “offline”.
1 | "probes": [ |
The trusted load balancer uses the same health probe (443 TCP), but the ports are different. Since we want to load balance all traffic, HA ports are used:
1 | "loadBalancingRules": [ |
In this case, “All” is specified to select all protocols (TCP and UDP). FrontendPort 0 and BackendPort 0 also means that all ports are selected.
An important setting in the network interfaces is the IP Forwarding. This enables the forwarding of traffic, not destined for the local network card.
1 | "properties": { |
The managed disks are imported from a given VHD file, uploaded in a previous post, and placed on blob storage. The location is specified using the blob uri.
1 | "properties": { |
In the firewall resources, a disk is attached as created beforehand.
1 | "osDisk": { |
Another important setting is to define a primary nic, since the firewalls have more than one nic.
1 | "networkInterfaces": [ |
We will also deploy a management virtual machine. The templates to deploy the management virtual machine can be found in the management folder on the github repository.
This template doesn’t contain any specialities. It is just a simple Windows VM, based on the 101-vm-simple-windows quickstart template.
For the deployment, we will use powershell.
1 | # Create the Resource groups for the resources |
Important notice
In both deployments, an extra parameter is specified. For the deployment of the firewalls, parameter pfSenseStorageAccountName is specified and for the management deployment, we have parameter adminPassword. The reason for these extra parameters is because we don’t want to store this sensitive information in the parameters file and therefore, we add them at runtime.
After both the firewalls and the management virtual machine are deployed, we need to perform some extra small, but important actions.
The most important action is to change the admin password of the pfSense appliances, if you didn’t change this in the image. They are reachable from the internet, so you don’t want to have a default password on the virtual appliances.
The next important actions is the routing to 168.63.129.16. This is a special IP address, used by Azure. It is the IP address of the DNS servers, but also used as the source IP address of the health probes of the load balancers. The part source IP address in this phrase is very important. Since Software Defined Networking is used, packets from an address can pop-up on an interface where needed. This is also the case for the health probes for the load balancers. For the untrusted load balancer, there is no issue. The ip 168.63.129.16 is considered a public IP and when checking the WAN ports on the firewalls, traffic is returned to the internet and it works. But for the trusted side of the load balancers, this is a different story. If no extra configuration is made, response to 168.63.129.16 on the trusted side will go through the firewall, being sent to the internet this way. This is asymmetric routing which won’t work.
To make this work, we need to perform these steps:
For the configuration of these settings, we can make use of the management virtual machine. Connect to the public IP of the virtual machine using RDP. The IPs of the firewalls are (if you didn’t change the ip addressing in the parameter files):
To create the gateway for the trusted subnet, navigate to System/Routing/Gateways:
Select the LAN interface for the gateway, give it a name and configure the IP address of the gateway of the subnet. In Azure, the first IP address of the subnet is the gateway. In this case it is 10.1.0.33. You cannot ping the gateway by design, so disable gateway monitoring and gateway monitoring action.
Apply the configuration.
Now, with the gateway in place, we can add the static route.
We want to create a static route to 168.63.129.16/32, point it to the gateway we created and give it a description.
First, we need to configure SSH in the advanced settings.
Now, we are able to putty to the LAN IPs of the pfSense instances. Login to the machines using SSH and select 8 in the menu to access bash.
In bash, enter the following command to delete the route:
1 | route delete 168.63.129.16 |
With the route deleted, traffic from 168.63.129.16, directly to the LAN port can be answered and the load balancer is coming online, as you can see in the monitoring of the load balancer:
Today, we did a lot of fun stuff! We created the firewalls with their corresponding load balancers. It would be great if I could deliver a totally automated deployment, but I really have no idea to block the static routes from DHCP. Still, we have a nice solution and in the next post, we will create the on-premise test environment.
]]>In the previous post, we designed the Azure Virtual Datacenter using the Hub-and-Spoke model. Now, it is time to get our hands dirty and start with the fun part! In this post, we will create the virtual networks that create the base
In this part we will create the foundation of the Azure Virtual Datacenter, the Virtual Networks. In these Virtual Networks, we will create subnets. These networks are separated network segments with different network addresses within the same Virtual Network “supernet”. Within the same Virtual Network, we don’t need to configure any routing. Software Defined Networking takes care of the routing. We will add User-Defined Routing later, but for now we use the default routing of the Azure software defined networking.
In our demo, we will create a Virtual Network with the prefix 10.1.0.0/16 in which we will create 4 subnets:
Network Security Group
A network security group is also needed to enable public load balancing. It is also made part of the template.
Two spoke Virtual Networks will be created, with a subnet that has the same size as the Virtual Network.
An on-premise Virtual Network is created with a Gateway Subnet for the Site-to-Site VPN and a subnet for performing tests.
We will create the networks using ARM templates. This method is idempotent, so it doesn’t matter how many times you deploy the template, the result will always be the same. Here an overview of what we will deploy in this part:
The templates can be found on my GitHub: https://github.com/christofvg/AzureVDC
To deploy the templates in azure, release pipelines can be used, but for now, PowerShell will be used. Since an Azure Virtual Datacenter Deployment is rather an advanced topic and I assume a brief knowledge on connecting to Azure using PowerShell and selecting the right subscription.
1 | # Login to Azure |
In this part, we layed the foundation of the lab, which is deployed in minutes. In the next part, we will deploy the firewalls, the load balancers for the firewalls and the management virtual machine for the firewalls.
]]>The term Azure Virtual Datacenter was introduced by Microsoft as an approach for extending your on-premises datacenter to the public cloud in a secure way. The complete description of the Azure Virtual Datacenter is described in the eBook “Azure Virtual Datacenter”, which can be found here: https://azure.microsoft.com/mediahandler/files/resourcefiles/1ad643b8-73f7-43f6-b05a-8e160168f9df/Azure_Virtual_Datacenter.pdf.
This series will cover the creation of an Azure Virtual Datacenter from a networking point-of-view. Software Defined Networking is a key component of the Azure Virtual Datacenter, which gives you a very flexible networking solution to work with. On the other hand, some things need to be done different, compared to traditional networking since layer 2 can’t be used in Software Defined Networking. In future series, I will cover governance and compliancy more in detail.
The demo environment I will create throughout the series will be an Azure Datacenter, based on a hub-and-spoke model. An on-premises network is also needed to test networking from and to the Azure Datacenter. I will create a separate Virtual Network to simulate the on-premises datacenter, connected with a site-to-site VPN.
Here an overview of the complete environment:
The environment is built on the Hub-and-Spoke model. In this model, there is a centralized hub that connects the external and spoke networks. All traffic is sent through centralized firewalls, enabling you to filter and analyze the network traffic. It also contains all necessary centralized services like management servers, Active Directory, DNS, … .
The workloads are separated in different spokes. These spokes are actually Virtual Networks, connected to the central hub using vNet peering. This way, you can separate different types of workloads delivering a secure environment for the services, running in the spokes. Another approach could be separating operational companies or teams in different spokes for segregation of duty.
User Defined Routing is one of the benefits of Software Defined Networking. Using User Defined Routes, you can steer traffic in a desired direction. In this case, we will send all traffic from the spokes and the gateway subnet to the load balancer of the centralized firewalls.
For internet traffic, traffic should use NAT, or more detailed sNAT or source NAT. This way, the source IP of a network package is translated to the public IP.
For traffic between east and west (between the on-premises network and the Azure Virtual Datacenter), routing should always be used. This is very important since some protocols, like Kerberos, don’t work (well) through NAT.
Before standard load balancers came into play, basic load balancers were the only way to use load balancing in Azure. But using a basic load balancer, ports need to be added one port per rule, where standard load balancers have HA ports where all ports can be specified in a single rule.
Since the standard load balancer came available, a true active-active firewall setup came in sight. Multiple ports (or the complete range of 1-65535) could be forwarded in a single rule, but traffic needs to be symmetric. If a load balancer is placed on the untrusted side, and another on the trusted side, all traffic is load balanced in both directions. But there is no guarantee that returning traffic will flow through the same firewall it came in. To solve this issue, firewall-on-a-stick is used where traffic flows in and out of the firewall through the same port. This means that only one load balancer is needed for both incoming and returning traffic.
The load balancer uses a hashing method to determine where to send the traffic to. This hashing method can be configured as a 2-tuple hash, or a 5-tuple hash. A 2-tuple hash creates a hash, based on the source IP and the destination IP. A 5-tuple hash create the hash based on the source and destination IPs, source and destination ports and the protocol. Using 5-tuple hash, session affinity is achieved.
In the following example, a packet is sent and returned through the firewalls. For the incoming flow, the hash is calculated on source IP 10.0.0.1 and destination IP 10.0.1.1, resulting in the example hash 123456789ABCDEF. When the recipient answers, the IPs are the same, but the order is reversed. The source IP is now 10.0.1.1 and the destination is now 10.0.0.1, resulting in the same hash and therefore, the same firewall is used.
More information about this topic can be found here: https://docs.microsoft.com/en-us/azure/load-balancer/load-balancer-ha-ports-overview
All examples will be deployed using ARM templates. This way, you could use them as a guide and implement them with minor changes if you want to try it yourself.
The ARM templates can be found on my Github: https://github.com/christofvg/AzureVDC
In this post we looked at the design of the Azure Virtual Datacenter. We covered the design decisions and explained the load balancing in the magical software defined network. In the next posts, we will make our hands dirty and start building our own Virtual Datacenter.
]]>After publishing part 1 through 3, someone brought to my attention that I should do the deployment using ARM templates instead of using PowerShell. This is completely true, so I created the necessary ARM templates to deploy the exact same environment. I made use of a main template that gathers all parameters and creates all resource groups. It also deploys 4 linked templates:
The code can be found on my Github account at https://github.com/christofvg/AzurePfSense.
To clone the repository:
1 | git clone https://github.com/christofvg/AzurePfSense.git |
The deployment is completely configured using the azuredeploy.parameters.json file. All parameters are available in that file except for 2 parameters that should be passed during the deployment as they should not be entered in source control.
To deploy the template using PowerShell:
1 | New-AzureRmDeployment -Location <location> -TemplateFile <path to azuredeploy.json> -TemplateParameterFile <path to azuredeploy.parameters.json> -vhdStorageAccountName <storageAccountName> -managementVmAdminPassword <password> -Verbose |
Using ARM templates, a deployment is done in a few minutes with a single command. Very suitable for deployments that need to be redeployed often. This is specially the case in lab environments.
]]>In the first part, we prepared the virtual machine for pfsense with all necessary tweeks for Azure. In part 2, all necessary packages are installed, along with the Azure Linux Agent. Now we are ready to upload the VHD to an Azure storage account, create an image and deploy a new virtual machine, based on that image.
I assume you have a storage account already in place.
To upload the VHD file, we will use the Microsoft Azure Storage Explorer.
First we need to deploy the Azure networks. We will deploy a Virtual network with the following properties:
Commands:1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16# Login to the Azure Account
Login-AzureRmAccount
# List all subscriptions, available onder your account
Get-AzureRmSubscription
# Select the right subscription
Select-AzureRmSubscription <subscription id>
# Create a new resource group
New-AzureRmResourceGroup -Name eu-network-rg -Location 'West Europe'
# Create a new Virtual Network
$VirtualNetwork = New-AzureRmVirtualNetwork -ResourceGroupName eu-network-rg -Location 'West Europe' -Name eu-vdc-vnet -AddressPrefix "10.0.0.0/24"
# Configure all subnets
Add-AzureRmVirtualNetworkSubnetConfig -Name gatewaysubnet -VirtualNetwork $VirtualNetwork -AddressPrefix "10.0.0.0/27"
Add-AzureRmVirtualNetworkSubnetConfig -Name frontend -VirtualNetwork $VirtualNetwork -AddressPrefix "10.0.0.32/27"
Add-AzureRmVirtualNetworkSubnetConfig -Name backend -VirtualNetwork $VirtualNetwork -AddressPrefix "10.0.0.64/27"
# Assign all subnets to the Virtual Network
$VirtualNetwork | Set-AzureRmVirtualNetwork
With the Virtual Networks in place, we can create the Managed Disk, based on the uploaded VHD. We will use the following properties:
Commands:1
2
3
4
5
6
7
8
9
10# Initialize variables
$storageType = "Standard_LRS"
$location = "West Europe"
$storageAccountId = "/subscriptions/<subscription id>/resourceGroups/RG_IMAGES_PFSENSE/providers/Microsoft.Storage/storageAccounts/azpfsense"
$sourceVhdUri = "https://<storage account name>.blob.core.windows.net/vhd/pfsense.vhd"
New-AzureRmResourceGroup -Name eu-firewalls-rg -Location 'West Europe'
# Create the disk configuration
$diskConfig = New-AzureRmDiskConfig -AccountType $storageType -Location $location -CreateOption Import -StorageAccountId $storageAccountId -SourceUri $sourceVhdUri
# Create the Managed Disk
New-AzureRmDisk -Disk $diskConfig -ResourceGroupName eu-firewalls-rg -DiskName eu-pfsense-1-os
We are ready to create the virtual machine for pfSense. We will create the virtual machine with the following properties:
Commands:1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16# Get the object of the existing Managed Disk
$disk = Get-AzureRmDisk -DiskName eu-pfsense-1-os -ResourceGroupName eu-firewalls-rg
# Get the object for the existing Virtual Network
$VirtualNetwork = Get-AzureRmVirtualNetwork -Name eu-vdc-vnet -ResourceGroupName eu-network-rg
# Create a new Virtual Machine object
$virtualMachine = New-AzureRmVMConfig -VMName eu-pfsense-1 -VMSize Standard_B2s
# Attach the existing Managed Disk to the Virtual Machine
$virtualMachine = Set-AzureRmVMOSDisk -VM $virtualMachine -ManagedDiskId $disk.Id -CreateOption Attach -Linux
# Create the NIC's for the frontend and the backend
$frontEndNic = New-AzureRmNetworkInterface -Name eu-pfsense-1-frontend-nic -ResourceGroupName eu-firewalls-rg -Location 'West Europe' -SubnetId $VirtualNetwork.Subnets[1].Id -PrivateIpAddress 10.0.0.36
$backEndNic = New-AzureRmNetworkInterface -Name eu-pfsense-1-backend-nic -ResourceGroupName eu-firewalls-rg -Location 'West Europe' -SubnetId $VirtualNetwork.Subnets[2].Id -PrivateIpAddress 10.0.0.68
# Add the NIC's to the Virtual Machine
$virtualMachine = Add-AzureRmVMNetworkInterface -VM $virtualMachine -Id $frontEndNic.Id -Primary
$virtualMachine = Add-AzureRmVMNetworkInterface -VM $virtualMachine -Id $backEndNic.Id
# Create the Virtual Machine
New-AzureRmVM -VM $virtualMachine -ResourceGroupName eu-firewalls-rg -Location 'West Europe'
To be able to test the configuration, we need a Virtual Machine in the backend subnet. The procedure is nearly the same as for the firewalls, but with some small differences. We will add a public IP to enable external access and we also need only one NIC. We will create the Virtual Machine with these properties:
Commands:1
2
3
4
5
6
7
8
9
10
11
12
13
14# Create the Public IP Address
$pip = New-AzureRmPublicIpAddress -ResourceGroupName eu-firewalls-rg -Location 'West Europe' -Name eu-pfsense-mgmt-pip -AllocationMethod Dynamic
# Create the NIC for the Virtual Machine
$nic = New-AzureRmNetworkInterface -Name eu-pfsense-mgmt-nic -ResourceGroupName eu-firewalls-rg -Location 'West Europe' -SubnetId $VirtualNetwork.Subnets[2].Id -PublicIpAddressId $pip.Id -PrivateIpAddress 10.0.0.37
# Create the Virtual Machine Object
$virtualMachine = New-AzureRmVMConfig -VMName eu-pfsense-mgmt -VMSize "Standard_B1s"
# Prepare the Virtual Machine for Windows
$virtualMachine | Set-AzureRmVMOperatingSystem -Windows -ComputerName eu-pfsense-mgmt -Credential (Get-Credential)
# Configure the Virtual Machine for Windows 2016 Datacenter
$virtualMachine | Set-AzureRmVMSourceImage -PublisherName MicrosoftWindowsServer -Offer WindowsServer -SKus 2016-Datacenter -Version latest
# Add the NIC to the Virtual Machine
$virtualMachine | Add-AzureRmVMNetworkInterface -Id $nic.Id
# Create the Virtual Machine
New-AzureRmVM -ResourceGroupName eu-firewalls-rg -Location 'West Europe' -VM $virtualMachine
With the Virtual Machine deployed, we can now access the deployed pfSense instance.
]]>In the previous part, we created the virtual machine that we will use to install pfSense on, with custom settings specific for Azure. As a recap, here an overview of the settings we made to make the virtual machine compatible with Azure:
We also added an extra NIC. This is not for compatibility with Azure, but it is necessary to configure pfSense as a router.
With everything set, we can install pfSense. When we boot the virtual machine, it will boot from the ISO file and should enter the installation menu.
The installation process is very straight-forward and basically next-next-finish. When finished, the installer asks to reboot. Don’t forget to remove the ISO from the DVD drive.
After the reboot, the pfSense wizard should start. We don’t need to setup any VLAN’s.
Now we have arrived to some very important settings. In Azure the first (primary) NIC will receive a gateway by DHCP. We need a gateway on the second NIC as well, but that will be configured later on manually. So, it is very important to configure hn0 as the WAN interface, and hn1 as the LAN interface.
The configuration will start now. When finished, the configuration menu is shown.
The WAN interface will receive a DHCP address by default, which is a good thing. But we need access to the LAN interface as well, since this is the only port where the web interface is accessible out-of-the-box.
All interfaces must be configured as DHCP client. But this cannot be done using this menu. So first we need to assign a temporary fixed address to the LAN port. Enter a fixed IP address using menu 2. In my case, I will use IP 10.0.0.123.
We don’t need a gateway on this interface for now and we don’t need IPv6 settings.
Now, the web interface is accessible at https://10.0.0.123/ (in my case).
We will use menu 14 now to enable SSH on pfSense.
Since we have an IP assigned to the LAN interface, it is accessible and SSH is installed, we can use Putty to do the rest of the configuration. By default the login is admin/pfsense.
Use menu 8 to enter the shell.
Some packages are required to function properly in Azure. Following packages are needed:
We also need the Azure Linux Agent, but is installed from GitHub instead of using a package.
Before we install the packages, it is recommended to upgrade the packaged software distribution.
1 | pkg upgrade |
To install bash, sudo and git:
1 | pkg install -y sudo bash git |
Python must be available as command as well, but by default the command to run Python is python2.7. To enable python as command, we need to add a symbolic link.
1 | ln -s /usr/local/bin/python2.7 /usr/local/bin/python |
With all packages installed, we can install the Linux Agent for Azure. The Agent is first cloned from the GitHub repository. Then, the latest version is selected using the tag in Git and the installation is performed using Python.
1 | # Clone the Git repository |
We also need to create a link to the waagent executable.
1 | ln -sf /usr/local/sbin/waagent /usr/sbin/waagent |
The last step in the preparation of pfSense is to configure the LAN interface as DHCP client. To do so, we need to browse to the LAN interface using a web browser. In my case, this is https://10.0.0.123.
The default login is the same as the SSH login (admin/pfsense).
The initial configuration process is very straight forward. Since you are creating your own default deployment, you can actually choose the settings you want.
There are 2 settings that are important during the initial configuration:
Now we are ready to configure the LAN interface. Go to menu Interfaces > LAN. Change “IPv4 Configuration Type” to DHCP and Save.
At this point, the configuration is finished and the image is prepared. In the pfSense menu, enter 6 to halt the system.
We configured the image so it can be deployed in Azure. Both the WAN and LAN interface are configured with a dynamic IP address, required packages are installed and the Linux Agent is installed from git.
In the next post, we will upload pfSense to the Azure Storage Account and deploy the image.
]]>In another series, I will build an Azure Virtual Datacenter. Central in the Virtual Datacenter design are the firewalls that will inspect and filter all traffic that passes through the central hub.
We will use pfSense firewalls in this series. pfSense provides very reasonable priced, enterprise grade NVA’s. Check the Azure Marketplace for all information about pfSense and pricing. A community edition is also available, which can be downloaded for free. This version is community supported on https://forum.netgate.com. For our study of networking in a Virtual Datacenter environment in Azure, the community edition will do fine.
In a production environment it is highly recommended to make use of the Netgate pfSense Firewall/VPN/Router from the Azure Marketplace, or subscribe for Netgate pfSense support. More info about a support subscription can be found here: https://www.netgate.com/support/.
To prepare the image, we will use Hyper-V. After the image is prepared, it will be uploaded to Azure. But before the image can be created, we first need to download the latest version of pfSense Community Edition. It can be found here: https://www.pfsense.org/download/. We need an 64 bit ISO image and it is recommended to take the mirror that is located the nearest to your location.
Our pfSense deployment needs 2 NIC’s that need to be accessible during preparation. So we need access to both ports. In Hyper-V, we need to configure a Virtual Switch with access from outside Hyper-V. This may be one single Virtual Switch, connected to one network. In a routing scenario, we certainly don’t want to connect an internal and an external port on the same network, but during setup, this is a practical solution.
Hyper-V creates virtual hard disks in VHDX format, which is not supported in Azure. We need to make sure, the virtual hard disk is created, exactly the way we want it. The requirements are:
With the virtual switch in place and the virtual hard disk created, we are ready to create the virtual machine for pfSense. Also for virtual machines, Azure has some requirements. So we need to address these requirements as well when creating the virtual machine. Here an overview:
When we configured the virtual machine, the summary is shown.
Some other settings need to be configured as well before pfSense can be installed. We need to add a network adapter with external access. The ISO of pfSense needs to be mounted on the DVD drive. The last setting, but very important, is the creation of checkpoints. This setting needs to be disabled as we need a VHD with everything in it. With checkpoints enabled, we have a VHD and a delta file with all changes in it. It is quite obvious that this is nothing we can work with.
Add network adapter:
Configure virtual switch:
Mount ISO to DVD drive:
Disable checkpoints:
We created the virtual machine, compatible with Azure. It is a generation 1 machine with a fixed size disk in VHD format. Checkpoints are disabled and we have 2 network cards.
In the next post, we will install pfSense on the virtual machine.
]]>