As Azure customers and partners build bigger and more complex solutions in their subscriptions, you might hit quota and throttling issues. These can be irksome and cause confusion. This article will walkthrough some of the scenarios I’ve seen and how to design with them in mind.
Let’s make sure we’re on the same page regarding terminology used in this article:
Azure Resource Manager (ARM) - The management layer and API behind all Azure resources
Resource Provider (RP) - Each resource type inside Azure has a RP which allows you to manage that resource (e.g. Storage, Key Vault, VMSS, etc.)
Quota - the maximum number of a specific resource available for your subscription. Similar to a credit card limit
Examples:
Subscription or Resource Quota
Max RPS for Storage account
Max size of single blob container
Azure Function default timeout
Maximum # of VMs in a VMSS
Some quotas have adjustable and non-adjustable quotas
Some adjustable quotas can be managed programmatically using the Azure Quota Service API
Throttling - maximum number of API requests you can make in a certain period. Similar to bandwidth throttling
NOTE: There are subscription and tenant level throttling limits. Each Storage, Networking, Compute and Azure Resource Graph also have throttling limits
NOTE: Throttling for RP’s are per subscription per region
Examples:
Rate limit of writes to a subscription per hour
Rate limit of Deleting a VMSS in 3 min
Managing Quotas
Because quotas are mostly static, viewing your quotas is pretty simple. Simply to go the Azure Portal and click on “My quotas”.
If you need to increase your quota, you might need to open an Azure Support ticket. For example, if you need to start deploying in a new region, you might need to open a ticket to increase the “Total Regional vCPUs” and “VMSS” quotas in “West Central US”. Once the ticket has been approved, the quota will be available to you.
Managing Throttling
For the most part, you won’t need to worry about throttling, but if you’re doing very large scale deployments with LOTS of constant churning of resources, you might hit throttling limits.
These limits are less about the number of resources, but HOW you use the resources. For example:
You can have 5000 AKS cluster in one subscription, each AKS cluster can have a maximum of 100 node pools. If you try creating the max # of AKS clusters with the max # of node pools simultaneously, then you’ll definitely hit the throttling limit.
Some OSS projects aggressively call ARM and the RP API’s in a reconciliation loop. Multiple instances of these projects will also hit the throttling limit.
Since throttling is specific to the current time window, it can be trickier. There’s no “hard formula” for when you’ll hit a threshold. But when you do, you’ll probably start seeing 429 HTTP status responses.
Throttling Examples
Thankfully, you can get insights into your current throttling status by looking at response headers for the requests.
x-ms-ratelimit-remaining-subscription-reads - # of read operations to this subscription remaining
x-ms-ratelimit-remaining-subscription-writes - # of writes operations to this subscription remaining
x-ms-ratelimit-remaining-resource - Compute RP specific header, which could show multiple policy statuses. (see “Read Request for GETting a VMSS” below for details)
Let’s dig into this deeper using the Azure CLI.
Example: Create a Resource Group (Write Request)
Because this request creates a RG, it will count against our subscription writes:
? az group create -n $RG --location $LOCATION --verbose --debug --debug 2>&1 | grep 'x-ms'
DEBUG: cli.azure.cli.core.sdk.policies: 'x-ms-client-request-id': ''
DEBUG: cli.azure.cli.core.sdk.policies: 'x-ms-ratelimit-remaining-subscription-writes': '1199'
DEBUG: cli.azure.cli.core.sdk.policies: 'x-ms-request-id': ''
DEBUG: cli.azure.cli.core.sdk.policies: 'x-ms-correlation-request-id': ''
DEBUG: cli.azure.cli.core.sdk.policies: 'x-ms-routing-request-id': 'SOUTHCENTRALUS:20230512T163152Z:'
NOTE: The key point is how the x-ms-ratelimit-remaining-subscription-writes is now 1199 (instead of the standard 1200 per hour as per the Subscription and Tenant limits)
Example: GET a VMSS (Read Request)
This request performs a GET (read) request on an existing VMSS. This is similar to the write request for the RG, but since Compute RP also has a separate set of throttling policies, it also counts against the Compute RP limits.
`markdown
? az vmss show -n $VMSS_NAME -g $RG --debug 2>&1 | grep x-ms
DEBUG: cli.azure.cli.core.sdk.policies: 'x-ms-client-request-id': ''
DEBUG: cli.azure.cli.core.sdk.policies: 'x-ms-ratelimit-remaining-resource': 'Microsoft.Compute/GetVMScaleSet3Min;197,Microsoft.Compute/GetVMScaleSet30Min;1297'
DEBUG: cli.azure.cli.core.sdk.policies: 'x-ms-request-id': ''
DEBUG: cli.azure.cli.core.sdk.policies: 'x-ms-ratelimit-remaining-subscription-reads': '11999'
DEBUG: cli.azure.cli.core.sdk.policies: 'x-ms-correlation-request-id': ''
DEBUG: cli.azure.cli.core.sdk.policies: 'x-ms-routing-request-id': 'SOUTHCENTRALUS:20230512T162738Z:'`
NOTE: The key point is how x-ms-ratelimit-remaining-resource has two key-value pairs:
Microsoft.Compute/GetVMScaleSet3Min;197 - I ran this command before, so I have 197 requests available in the 3 minute window for performing GET requests on the VMSS resource
Microsoft.Compute/GetVMScaleSet30Min;1297 - I now have 1297 requests available in the 30 minute window for performing GET requests on VMSS resources
NOTE: x-ms-ratelimit-remaining-subscription-reads doesn’t seem to decrease (11999). Even if I run the same command again. I haven’t figured that out yet.
Designing with quotas and throttling in mind
Most Azure deployments won’t need this type of fine tuning, but just in case, there’s some documented Throttling Best Practices as well as my personal pro-tips:
Use the Azure SDK, as many services have the recommended retry guidance built-in
Instead of creating and deleting VMSS (which consume multiple VMSS API requests), scale the VMSS to 0 (which only consumes 1 VMSS API request)
Any type of Kubernetes cluster auto-scaler will perform a reconciliation loop with Azure Compute RP. This could eat into your throttling limits
Use the Azure Quota Service API to programmatically request quota increases
If you’re unable to workaround the throttling limits, then the next step is to look at the Deployment Stamp pattern using multiple subscriptions. You can programmatically create subscriptions using Subscription vending.
Hopefully this article has helped you understand quotas limits and throttling limits in Azure, and how to work around them. Let me know if you have any additional questions and/or feedback and I can follow-up with additional details.
This walkthrough shows how to setup a Private Link Service with an AKS cluster and create a Private Endpoint in a separate Vnet.
While many tutorials might give you a full ARM template, this is designed as a walkthrough which completely uses the CLI so you can understand what's happening at every step of the process.
It focuses on an "uninteresting" workload and uses podinfo as the sample app. This is because it's easy to deploy and customize with a sample Helm chart.
This is inspired and leans heavily on the Azure Docs for creating a Private Link Service.
Architecture
Private Link Endpoint Service
Prerequisites
Azure CLI
jq
Assumptions
This walkthrough assumes you let Azure create the Vnet when creating the AKS cluster. If you manually created the Vnet, then the general steps are the same, except you must enter the AKS_MC_VNET, AKS_MC_SUBNET env vars manually.
Setup Steps
First, create a sample AKS cluster and install Podinfo on it.
Set these values
AKS_NAME=
AKS_RG=
LOCATION=
Create the AKS cluster
az aks create -n $AKS_NAME -g $AKS_RG
Get the MC Resource Group
AKS_MC_RG=$(az aks show -n $AKS_NAME -g $AKS_RG | jq -r '.nodeResourceGroup')
echo $AKS_MC_RG
Get the Vnet Name
AKS_MC_VNET=$(az network vnet list -g $AKS_MC_RG | jq -r '.[0].name')
echo $AKS_MC_VNET
AKS_MC_SUBNET=$(az network vnet subnet list -g $AKS_MC_RG --vnet-name $AKS_MC_VNET | jq -r '.[0].name')
echo $AKS_MC_SUBNET
AKS_MC_LB_INTERNAL=kubernetes-internal
AKS_MC_LB_INTERNAL_FE_CONFIG=$(az network lb rule list -g $AKS_MC_RG --lb-name=$AKS_MC_LB_INTERNAL | jq -r '.[0].frontendIpConfiguration.id')
echo $AKS_MC_LB_INTERNAL_FE_CONFIG
Deploy a sample app using an Internal LB
helm upgrade --install --wait podinfo-internal-lb \
--set-string service.annotations."service\.beta\.kubernetes\.io\/azure-load-balancer-internal"=true \
--set service.type=LoadBalancer \
--set ui.message=podinfo-internal-lb \
podinfo/podinfo
Install Steps - Create the Private Link Service
These steps will be done in the MC_ resource group.
Disable the private link service network policies
az network vnet subnet update \
--name $AKS_MC_SUBNET \
--resource-group $AKS_MC_RG \
--vnet-name $AKS_MC_VNET \
--disable-private-link-service-network-policies true
Create the PLS
PLS_NAME=aks-pls
az network private-link-service create \
--resource-group $AKS_MC_RG \
--name $PLS_NAME \
--vnet-name $AKS_MC_VNET \
--subnet $AKS_MC_SUBNET \
--lb-name $AKS_MC_LB_INTERNAL \
--lb-frontend-ip-configs $AKS_MC_LB_INTERNAL_FE_CONFIG
Install Steps - Create the Private Endpoint
These steps will be done in our private-endpoint-rg resource group.
PE_RG=private-endpoint-rg
az group create \
--name $PE_RG \
--location $LOCATION
PE_VNET=pe-vnet
PE_SUBNET=pe-subnet
az network vnet create \
--resource-group $PE_RG \
--name $PE_VNET \
--address-prefixes 10.0.0.0/16 \
--subnet-name $PE_SUBNET \
--subnet-prefixes 10.0.0.0/24
Disable the private link service network policies
az network vnet subnet update \
--name $PE_SUBNET \
--resource-group $PE_RG \
--vnet-name $PE_VNET \
--disable-private-endpoint-network-policies true
PE_CONN_NAME=pe-conn
PE_NAME=pe
az network private-endpoint create \
--connection-name $PE_CONN_NAME \
--name $PE_NAME \
--private-connection-resource-id $PLS_ID \
--resource-group $PE_RG \
--subnet $PE_SUBNET \
--manual-request false \
--vnet-name $PE_VNET
We need the NIC ID to get the newly created Private IP
PE_NIC_ID=$(az network private-endpoint show -g $PE_RG --name $PE_NAME -o json | jq -r '.networkInterfaces[0].id')
echo $PE_NIC_ID
Get the Private IP from the NIC
PE_IP=$(az network nic show --ids $PE_NIC_ID -o json | jq -r '.ipConfigurations[0].privateIpAddress')
echo $PE_IP
Validation Steps - Create a VM
Lastly, validate that this works by creating a VM in the Vnet with the Private Endpoint.
VM_NAME=ubuntu
az vm create \
--resource-group $PE_RG \
--name ubuntu \
--image UbuntuLTS \
--public-ip-sku Standard \
--vnet-name $PE_VNET \
--subnet $PE_SUBNET \
--admin-username $USER \
--ssh-key-values ~/.ssh/id_rsa.pub
VM_PIP=$(az vm list-ip-addresses -g $PE_RG -n $VM_NAME | jq -r '.[0].virtualMachine.network.publicIpAddresses[0].ipAddress')
echo $VM_PIP
SSH into the host
ssh $VM_IP
$ curl COPY_THE_VALUE_FROM_PE_IP:9898
The output should look like:
$ curl 10.0.0.5:9898
{
"hostname": "podinfo-6ff68cbf88-cxcvv",
"version": "6.0.3",
"revision": "",
"color": "#34577c",
"logo": "/images/2022/cuddle_clap.gif",
"message": "podinfo-internal-lb",
"goos": "linux",
"goarch": "amd64",
"runtime": "go1.16.9",
"num_goroutine": "9",
"num_cpu": "2"
}
Multiple PLS/PE
To test a specific use case, I wanted to create multiple PLS and PE's. This set of instructions lets you easily loop through and create multiple instances.
podinfo requires a high numbered port, eg 9000+
SUFFIX=9000
helm upgrade --install --wait podinfo-$SUFFIX \
--set-string service.annotations."service\.beta\.kubernetes\.io\/azure-load-balancer-internal"=true \
--set service.type=LoadBalancer \
--set service.httpPort=$SUFFIX \
--set service.externalPort=$SUFFIX \
--set ui.message=podinfo-$SUFFIX \
podinfo/podinfo
This might be easier to hard-code
AKS_MC_LB_INTERNAL_FE_CONFIG=$(az network lb rule list -g $AKS_MC_RG --lb-name=$AKS_MC_LB_INTERNAL -o json | jq -r ".[] | select( .backendPort == $SUFFIX) | .frontendIpConfiguration.id")
echo $AKS_MC_LB_INTERNAL_FE_CONFIG
PLS_NAME=aks-pls-$SUFFIX
PE_CONN_NAME=pe-conn-$SUFFIX
PE_NAME=pe-$SUFFIX
az network private-link-service create \
--resource-group $AKS_MC_RG \
--name $PLS_NAME \
--vnet-name $AKS_MC_VNET \
--subnet $AKS_MC_SUBNET \
--lb-name $AKS_MC_LB_INTERNAL \
--lb-frontend-ip-configs $AKS_MC_LB_INTERNAL_FE_CONFIG
PLS_ID=$(az network private-link-service show \
--name $PLS_NAME \
--resource-group $AKS_MC_RG \
--query id \
--output tsv)
echo $PLS_ID
az network private-endpoint create \
--connection-name $PE_CONN_NAME \
--name $PE_NAME \
--private-connection-resource-id $PLS_ID \
--resource-group $PE_RG \
--subnet $PE_SUBNET \
--manual-request false \
--vnet-name $PE_VNET
PE_NIC_ID=$(az network private-endpoint show -g $PE_RG --name $PE_NAME -o json | jq -r '.networkInterfaces[0].id')
echo $PE_NIC_ID
PE_IP=$(az network nic show --ids $PE_NIC_ID -o json | jq -r '.ipConfigurations[0].privateIpAddress')
echo $PE_IP
echo "From your Private Endpoint VM run: curl $PE_IP:$SUFFIX"
I created this article to help myself (and hopefully you!) to clearly understand all of the resources and how they interact to create a Private Link Service and Private Endpoint fronting a private service inside an AKS cluster. This has been highly enlightening for me and I hope it has for you too.
This article explains why, when, and how to use self-managed Kubernetes clusters in Azure for testing custom scenarios.
Kubernetes has gotten so large and complex that most companies prefer to use the managed service (e.g. AKS, GKE) instead of running it themselves. By using a managed Kubernetes service, this frees up the operations team to focus on their core competency instead of optimizing, backing up and upgrading of Kubernetes.
While this reduces the operational burden, you lose the ability to modify the platform. Sometimes these are acceptable tradeoffs, sometimes you need to manage it yourself.
Historically, AKS-engine was the OSS tool for creating unmanaged Kubernetes clusters on Azure, but it had some limitations. CAPI/CAPZ is the go-forward solution for creating and operating self-managed clusters declaratively.
I highly recommend reading Scott Lowe's article on An introduction to CAPI. It covers a lot of terminology and concepts used here.
One of the reasons for using CAPI/CAPZ is as a testing and development tool for Kubernetes on Azure. For example, you might need to build and test the following scenarios:
A kernel change to the worker nodes
A modification to the K8S config on control plane nodes
An installation of a different CNI
The use of K8S to manage K8S
This diagram represents a high level architecture of a starter CAPI/CAPZ cluster.
The rest of this article will explain how to implement the above scenarios utilizing the CAPI quickstart. Because the command arguments will change over time, this article will describe the steps and provide a link to the full details like this:
Link to CAPI Quick Start with details: base command to run
Create the KIND Cluster
Similar to RepRap, CAPI uses a Kubernetes cluster to make more Kubernetes clusters. The easiest way is with Kuberenetes IN Docker (KIND). As the name implies, it's a Kubernetes cluster which runs as a Docker container. This is our starting point for what we call "Bootstrap Cluster".
Create Kind Cluster: kind create cluster
Initialize cluster for Azure
We will use this bootstrap cluster to initialize the "Management Cluster" which contains all of the CRDs and runs the CAPI controllers. This is where we will apply all of our changes to meet our scenarios.
Initialize cluster for Azure: clusterctl init --infrastructure azure
Generate cluster configuration
Now that our management cluster is ready, we want to define what our workload cluster will look like. Thankfully, there are different flavors we can pick from. By using the default, we will get an unmanaged K8S cluster using virtual machines.
Generate cluster configuration: clusterctl generate cluster capi-quickstart > capi-quickstart.yaml
We now have a file which contains the CRDs which will define our workload cluster. We will modify capi-quickstart.yaml and edit the CRDs to implement each of our scenarios.
Full documentation is available for CAPI (baseline) CRDs and CAPZ (Azure specific resources) CRDs.
Scenario: Worker node kernel change
If we want to modify the worker nodes, we likely want to add a preKubeadmCommands and postKubeadmCommands directive in the KubeadmConfigTemplate.
preKubeadmCommands allows a list of commands to run on the worker node BEFORE joining the cluster.
postKubeadmCommands allows a list of commands to run on the worker node AFTER joining the cluster.
apiVersion: bootstrap.cluster.x-k8s.io/v1alpha4
kind: KubeadmConfigTemplate
metadata:
name: capi-quickstart-md-0
namespace: default
spec:
template:
spec:
preKubeadmCommands:
wget -P /tmp https://kernel.ubuntu.com/.deb
dpkg -i /tmp/.deb
postKubeadmCommands:
reboot
After you've made these changes, you can proceed to the rest of the steps by applying the resources to your management cluster which will then create your workload cluster and deploy the CNI.
Scenario: Modify Kubernetes components
If we want to modify the control plane, we can make changes to the KubeadmControlPlane. This allows us to leverage the kubeadm API to customize various components.
For example, to enable a Feature Gate on the kube-apiserver:
apiVersion: controlplane.cluster.x-k8s.io/v1alpha4
kind: KubeadmControlPlane
metadata:
name: capi-quickstart-control-plane
namespace: default
spec:
kubeadmConfigSpec:
clusterConfiguration:
apiServer:
extraArgs:
feature-gates: MyFeatureGate=true
The above example omits some fields for brevity. Make sure that you keep any existing args and configurations that you are not modifying in-place.
After you've made these changes, you can proceed to the rest of the steps by applying the resources to your management cluster which will then create your workload cluster and deploy the CNI.
Apply the Workload Cluster
Now that we have defined what our cluster should look like, apply the resources to the management cluster. The CAPZ operator will detect the updated resources and talk to Azure Resource Manager.
Apply the workload cluster kubectl apply -f capi-quickstart.yaml
Monitor the Cluster Creation
After you've made the changes to the capi-quickstart.yaml resources and applied them, you're ready to watch the cluster come up.
Watch the cluster creation:
kubectl get cluster
clusterctl describe cluster capi-quickstart
kubectl get kubeadmcontrolplane - Verify the Control Plane is up
Now that the workload cluster is up and running, it's time to start using it!
Get the Kubeconfig for the Workload Cluster
Now that we're dealing with two clusters (management cluster in Docker and workload cluster in Azure), we now have two kubeconfig files. For ease, we will save it to the local directory.
Get the Kubeconfig for the workload cluster clusterctl get kubeconfig capi-quickstart > capi-quickstart.kubeconfig
Install the CNI
By default, the workload cluster will not have a CNI and one must be installed.
Deploy the CNI kubectl --kubeconfig=./capi-quickstart.kubeconfig apply -f https://...calico.yaml
Scenario: Install a different CNI
If you want to use flannel as your CNI, then you can apply the resources to your management cluster which will then create your workload cluster.
However, instead of Deploying the CNI, you can follow the steps in the Install Flannel walkthrough.
Cleanup
When you're done, you can cleanup both the workload and management cluster easily.
Delete the workload cluster kubectl delete cluster capi-quickstart
If you want to create the workload cluster again, you can do so by re-applying capi-quickstart.yaml
Delete the management cluster kind delete cluster
If you want to create the management cluster again, you must start from scratch. If you delete the management cluster without deleting the workload cluster, then the workload cluster and Azure resources will remain.
Summary
Similar to how Kubernetes allows you to orchestrate containers using a declarative syntax, CAPI/CAPZ allows you to do the same, but for Kubernetes clusters in Azure.
This article covered example scenarios for when to use CAPI/CAPZ as well as a walkthrough on how to implement them.
I'm especially excited for the future of CAPI/CAPZ and how it can integrate with other Cloud Native methodologies like GitOps to declaratively manage clusters.
P.S. I am extremely grateful to Cecile Robert Michon's (Twitter & Github) technical guidance for this article. Without her support, I wouldn't have gotten this far and definitely would have missed a few key scenarios. Thanks Cecile!
Azure Functions (one of the many Serverless Platforms inside Azure) allows you to use Python as your runtime environment.
This is great; however, it requires Python 3.6, which is a problem for my development on a Mac (which uses Python 3.7). Downgrading/dual installation has the potential for many perils, so I wanted an isolated runtime development environment which wouldn't interfere with my current setup.
Here's my requirements:
Run Azure Functions locally (e.g. "func host start")
Publish Azure Functions locally (e.g. "func azure functionapp publish")
Use my existing Azure CLI session (i.e. don't have to login each time)
Won't confuse my existing Python 3.7 env.
Docker to the rescue!
I'll spare you the details of the iterative process of creating the Dockerfile, but after a some iterations, I knew I was on the right track.
You can copy/create my code here:
https://gist.github.com/lastcoolnameleft/05b6b09735fb435b2cb4469f6cf30ac6
In short, it creates a Docker image and runs it with:
Ubuntu 18.04
Python 3.6
Azure CLI
Azure Function Core Tools
Forwards port 7071 from inside the container to my localhost (used for local function testing)
Mounts my home dir to /root to maintain my Azure CLI session. (e.g. No login!)
This will definitely save me time each time I want to setup a new Function (or other) environment and I hope it helps save time for you too.
Make my func the p-func!
The internet’s a weird place. We already knew that, yet it keeps finding new ways to amaze me. Someone thought it would be a novel idea to incorporate…
As much as Cloud Providers tout their availability and uptime, disasters happen. It's inevitable. And it's usually up to you to be prepared. There are services that can help; however, they're not always "Kubernetes aware".
Thankfully, the great folks at Heptio open-sourced Ark, a Disaster Recovery tool which works for all the major cloud providers.
I got hands-on with Ark and followed their Azure steps. It was a good start, but didn't highlight how an actual failover and recovery would look to the operator. I took their steps and created a step-by-step guide to perform a full migration.
Ark support Azure native resources, namely Managed Disk + Snapshots.
You can review those steps here: https://github.com/heptio/ark/blob/master/docs/azure-config.md
Another option would be to use Restic, which performs backups to a local file system. Later, I'll detail the steps on how to use Restic with Azure.
If you're looking for Best Practices on supporting Business Continuity and Disaster Recovery for AKS/K8S clusters in Azure, you're in luck! I wrote a Microsoft article covering this use case, which can be found here:
https://docs.microsoft.com/en-us/azure/aks/operator-best-practices-multi-region
WARNING: SSH'ing into an agent node is an anti-pattern and should be avoided. However, we don't live in an ideal world, and sometimes we have to do the needful.
Overview
This walkthrough creates an SSH Server running as a Pod in your Kubernetes cluster and uses it as a jumpbox to the agent nodes. It is designed for users managing a Kubernetes cluster who cannot readily SSH to into their agent nodes (e.g. AKS) does not publicly expose the agent nodes for security considerations).
This is one of the steps in the Kubernetes Workshop I have built when working with our partners.
NOTE
It has been tested in AKS cluster; however, it should also work in other cloud providers.
You can follow the steps on the SSH to AKS Cluster Nodes walkthrough; however, that requires you to upload your Private SSH key which I would rather avoid.
Assumptions
The SSH Public key has been installed for your user on the Agent host
You have jq installed Not vital, but makes the last step easier to understand.
Install an SSH Server
If you're paranoid, you can generate your own SSH server container; however, (https://github.com/corbinu/ssh-server) has some pretty good security defaults and is available on Docker Hub.
kubectl run ssh-server --image=corbinu/ssh-server --port=22 --restart=Never
Setup port forward
Instead of exposing a service with an IP+Port, we'll take the easy way and use kubectl to port-forward to your localhost.
NOTE: Run this in a separate window since it will need to be running for as long as you want the SSH connection
kubectl port-forward ssh-server 2222:22
Inject your Public SSH key
Since we're using the ssh-server as a jumphost, we need to inject our SSH key into the SSH Server. Using root for simplicity's sake, but I recommend a more secure approach going forward. (TODO: Change this to use a non-privileged user.)
cat ~/.ssh/id_rsa.pub | kubectl exec -i ssh-server -- /bin/bash -c "cat >> /root/.ssh/authorized_keys"
SSH to the proxied port
Using the SSH Server as a jumphost (via port-forward proxy), ssh into the IP address of the desired host.
`# Get the list of Host + IP's
kubectl get nodes -o json | jq '.items[].status.addresses[].address'
$USER = Username on the agent host
$IP = IP of the agent host
ssh -J root@127.0.0.1:2222 $USER@$IP`
NOTE: If you get "WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!" You might need to add -o StrictHostKeyChecking=no to the SSH command if you bounce across clusters. This is because SSH believes that the identity of the host has changed and you need to either remove that entry from your ~/.ssh/known_hosts or tell it to ignore the host identity.
Cleanup
kubectl delete pod ssh-server
Kill the kubectl port-forward command
Azure App Service for Linux is a pretty neat offering from Azure. You get all of the DevOps features you want (A/B Testing, Hosted Application, Tiered Support, Button-click scaling, lots of templates and more!) without the headache of managing VM's.
9 years ago, I wrote a quacky little website called "Duckiehunt". Unfortunately, I didn't pay the tech debt and things kept breaking until it was abandoned. I'm now using Duckiehunt as a learning ground for Azure's services and alternatives.
Azure App Service for Linux was the perfect fit. However, back in 2008 SSL wasn't as ubiquitous. Now, it's a badge of shame to NOT have it. Azure does offer an App Service Certificate, but I'd like to find a cheaper/more open solution.
Enter Let'sEncrypt from Mozilla and the EFF. If you don't know, EFF are the unsung heroes of the internet. They fight tirelessly to support your freedom and rights on the internet. Mozilla and EFF offer Let'sEncrypt as a free way to encrypt websites via CertBot. Now I'll dig into the technical details behind encrypting an App Service for Linux with Let'sEncrypt.
Step #1: Get CertBot
Because I'm on OSX, I was able to run: brew install certbot. For the full range of options, CertBot's webpage has what you need.
Step #2: Create Cert locally
Before CertBot can create the certificate for you, it must first validate you own the domain. It will prompt you for a few questions, and then ask you to create a file on the webhost and add content to that file for validation.
Thankfully, Azure App Service for Linux provides a terminal access to your container so you can make these modifications yourself.
➜ sudo certbot certonly -d duckiehunt.com --manual
...
Create a file containing just this data:
%RANDOM STRING 1%
And make it available on your web server at this URL:
http://duckiehunt.com/.well-known/acme-challenge/%RANDOM STRING 2%
-------------------------------------------------------------------------------
Press Enter to Continue
Step #3: Add the validation file to you website
I then went to the Kudu instance of my App Service and ran:
➜ mkdir /var/www/html/.well-known/acme-challenge/
➜ echo "%RANDOM STRING 2%" > %RANDOM STRING 1%
At this point, the validation is in place and it's time to continue with Chatbot by pressing "Enter".
Waiting for verification...
Cleaning up challenges
IMPORTANT NOTES:
- Congratulations! Your certificate and chain have been saved at:
/etc/letsencrypt/live/duckiehunt.com/fullchain.pem
Your key file has been saved at:
/etc/letsencrypt/live/duckiehunt.com/privkey.pem
Your cert will expire on 2017-11-12. To obtain a new or tweaked
version of this certificate in the future, simply run certbot
again. To non-interactively renew all of your certificates, run
"certbot renew"
- If you like Certbot, please consider supporting our work by:
Donating to ISRG / Let's Encrypt: https://letsencrypt.org/donate
Donating to EFF: https://eff.org/donate-le
Huzzah! I've now got a certificate. Time to upload.
Step #4: Upload the certificate to Azure
Azure has a pretty descriptive set of steps for associating a certificate to your App Service, which I was able to follow.
Openssl will ask for a Password which you need to keep as you upload the cert to Azure.
➜ cd /etc/letsencrypt/live/duckiehunt.com
➜ openssl pkcs12 -export -out myserver.pfx -inkey privkey.pem -in fullchain.pem
Enter Export Password:
Verifying - Enter Export Password:
➜ cp myserver.pfx ~/Desktop
Step #5: Bind the certificate to your App Service
From here on you're ready to Bind your SSL Certificate to your App Service. I'll let Microsoft's documentation lead the way from here.
https://docs.microsoft.com/en-us/azure/app-service-web/app-service-web-tutorial-custom-ssl#bind-your-ssl-certificate
Step #6: Bask in doing your part to secure the internet.
In summary, the process was pretty painless.
I used Let'sEncrypt to create a new Certificate for my App Service for Linux by creating a file that Let'sEncrypt could use to validate I owned the site.
I then encrypted that certificate to upload to Azure.
Once it was uploaded, I bound that certificate to my domain and voila! A more secure Duckiehunt
One bummer is that the certificate is intended to expire in 3 months instead of the industry standard of 12 months. The renewal process looks pretty easy, but that's a different blog post.
--Tommy feels that he's done his part in making the world a bit safer.
Like most children of the 80's, I loved playing with LEGO. By mixing and matching bricks, you could physically manifest your imagination.
My first LEGO set was the Blacktron - Renegade.
Blacktron Renegade
By following the instructions, I was able to explore space and move strange and dangerous cargo from distant planets. By moving the wings around, I was able to make the Batwing and fly around Gotham. (Well before anyone else realized that potential.)
This was an immensely rewarding experience that I've carried with me through my professional career.
Naturally, the toys of the child lead us to adulthood. I knew I wanted to spend my life building. Creating. Spawning new ideas. I wanted to physically manifest my ideas into structures that others would see, admire and even work/play/live in. When I learned that you could get a job doing this, I was elated. I knew this was exactly what I wanted to do. My mission in life was set.
One fateful day, when I was sharing my new life mission with my Godmother she informed me: "To be an architect you have to know how to draw." Anyone who's seen me sign a check, write on a whiteboard, or even attempt to draw a square knows artistry genes were not bestowed upon me. I was crushed. My life's mission was aborted and I was unsure what to do with myself.
My first drawing of the Falgout Family (I ran out of time for arms)
To quote my wife: "Those are people? I thought those were windows..."
I drew this
I drew this. Not sure what my obsession with blue people was. That drawing is nightmare fuel for me.
In High School, when Career Day came I didn't care about any session other than the local architect. As torturous as it was, I still wanted to know what it was like. All I remember was "hard work...something something...dedication".
Fast forward to the last 12 months. I made an exciting and brave leap to join Microsoft, and am now a "Cloud Solution Architect". I'm an Architect. I'm a real, bonafide Architect. (I'm literally crying as I write this as I'm so overwhelmed with a sense of accomplishment.) My bricks aren't 8x8x9.6 mm, they're CPU Cores. I no longer have one toychest, I have 36 datacenter regions, spanned across the world.
Thankfully, I'm not planning to give up on those plastic pieces of creativity, as I've currently got a Star Destroyer hanging from the ceiling of my man cave. And even more sets left to complete.
LEGO Star Destroyer hanging from the ceiling.
If I could go back and comfort my younger self during that heartbreaking moment, I'm sure I would have told him: "hard work...something something...dedication".
//build is a developer-centric conference Microsoft hosts every year. Since I never expected to work for Microsoft, I wasn't even aware of //build. So,…
Laziness, Impatience, Hubris
The great Larry Wall claims that these are the three great virtues of a great programmer. And I whole heartedly agree. However, If I were to propose three virtues, they’d be: Inquisitiveness, Acceptance, and Stubbornness.
My name is Tommy Falgout, I’m a new employee at Microsoft and I have no idea what I’m doing.
I don’t know .NET. Or Azure. I don’t own a Windows Phone. Heck, the last Windows OS I “owned†was XP.
What I do have is ~20 years of experience in *nix and Open Source software development. I helped develop the original SMS implementation for GSM, back when phones were only meant for voice. I wrote telecom automation systems in Perl, PHP and MySQL 3.x. I then worked at Yahoo for 9 years where I expanded my brain to build their live events engine to broadcast Obama’s Inauguration, the Royal Wedding and the NFL games (Yahoo used to own NFL streaming rights in the 2000’s) I migrated to Yahoo’s Infrastructure database which was the duct tape keeping everything together and integrated.
As a Technical Evangelist, I’m building upon all that experience as a foundation for this new opportunity. To take Azure to the next level.
The thing is...I don’t know how to Azure yet. But that’s the point of this blog. To detail my findings and explore the union of Open Source and Azure. This will be a dumping ground and lesson’s learned. I’m a big fan of transparency and learning from other people’s mistakes. My hope is that you can learn from mine.
Now to go learn how not to mangle my ARM.
