Skip to content

Tag: devops

Managing Azure Subscription Quota and Throttling Issues

As Azure customers and partners build bigger and more complex solutions in their subscriptions, you might hit quota and throttling issues. These can be irksome and cause confusion. This article will walkthrough some of the scenarios I’ve seen and how to design with them in mind. Let’s make sure we’re on the same page regarding terminology used in this article: Azure Resource Manager (ARM) - The management layer and API behind all Azure resources Resource Provider (RP) - Each resource type inside Azure has a RP which allows you to manage that resource (e.g. Storage, Key Vault, VMSS, etc.) Quota - the maximum number of a specific resource available for your subscription. Similar to a credit card limit Examples: Subscription or Resource Quota Max RPS for Storage account Max size of single blob container Azure Function default timeout Maximum # of VMs in a VMSS Some quotas have adjustable and non-adjustable quotas Some adjustable quotas can be managed programmatically using the Azure Quota Service API Throttling - maximum number of API requests you can make in a certain period. Similar to bandwidth throttling NOTE: There are subscription and tenant level throttling limits. Each Storage, Networking, Compute and Azure Resource Graph also have throttling limits NOTE: Throttling for RP’s are per subscription per region Examples: Rate limit of writes to a subscription per hour Rate limit of Deleting a VMSS in 3 min Managing Quotas Because quotas are mostly static, viewing your quotas is pretty simple. Simply to go the Azure Portal and click on “My quotas”. If you need to increase your quota, you might need to open an Azure Support ticket. For example, if you need to start deploying in a new region, you might need to open a ticket to increase the “Total Regional vCPUs” and “VMSS” quotas in “West Central US”. Once the ticket has been approved, the quota will be available to you. Managing Throttling For the most part, you won’t need to worry about throttling, but if you’re doing very large scale deployments with LOTS of constant churning of resources, you might hit throttling limits. These limits are less about the number of resources, but HOW you use the resources. For example: You can have 5000 AKS cluster in one subscription, each AKS cluster can have a maximum of 100 node pools. If you try creating the max # of AKS clusters with the max # of node pools simultaneously, then you’ll definitely hit the throttling limit. Some OSS projects aggressively call ARM and the RP API’s in a reconciliation loop. Multiple instances of these projects will also hit the throttling limit. Since throttling is specific to the current time window, it can be trickier. There’s no “hard formula” for when you’ll hit a threshold. But when you do, you’ll probably start seeing 429 HTTP status responses. Throttling Examples Thankfully, you can get insights into your current throttling status by looking at response headers for the requests. x-ms-ratelimit-remaining-subscription-reads - # of read operations to this subscription remaining x-ms-ratelimit-remaining-subscription-writes - # of writes operations to this subscription remaining x-ms-ratelimit-remaining-resource - Compute RP specific header, which could show multiple policy statuses. (see “Read Request for GETting a VMSS” below for details) Let’s dig into this deeper using the Azure CLI. Example: Create a Resource Group (Write Request) Because this request creates a RG, it will count against our subscription writes: ? az group create -n $RG --location $LOCATION --verbose --debug --debug 2>&1 | grep 'x-ms' DEBUG: cli.azure.cli.core.sdk.policies: 'x-ms-client-request-id': '' DEBUG: cli.azure.cli.core.sdk.policies: 'x-ms-ratelimit-remaining-subscription-writes': '1199' DEBUG: cli.azure.cli.core.sdk.policies: 'x-ms-request-id': '' DEBUG: cli.azure.cli.core.sdk.policies: 'x-ms-correlation-request-id': '' DEBUG: cli.azure.cli.core.sdk.policies: 'x-ms-routing-request-id': 'SOUTHCENTRALUS:20230512T163152Z:' NOTE: The key point is how the x-ms-ratelimit-remaining-subscription-writes is now 1199 (instead of the standard 1200 per hour as per the Subscription and Tenant limits) Example: GET a VMSS (Read Request) This request performs a GET (read) request on an existing VMSS. This is similar to the write request for the RG, but since Compute RP also has a separate set of throttling policies, it also counts against the Compute RP limits. `markdown ? az vmss show -n $VMSS_NAME -g $RG --debug 2>&1 | grep x-ms DEBUG: cli.azure.cli.core.sdk.policies: 'x-ms-client-request-id': '' DEBUG: cli.azure.cli.core.sdk.policies: 'x-ms-ratelimit-remaining-resource': 'Microsoft.Compute/GetVMScaleSet3Min;197,Microsoft.Compute/GetVMScaleSet30Min;1297' DEBUG: cli.azure.cli.core.sdk.policies: 'x-ms-request-id': '' DEBUG: cli.azure.cli.core.sdk.policies: 'x-ms-ratelimit-remaining-subscription-reads': '11999' DEBUG: cli.azure.cli.core.sdk.policies: 'x-ms-correlation-request-id': '' DEBUG: cli.azure.cli.core.sdk.policies: 'x-ms-routing-request-id': 'SOUTHCENTRALUS:20230512T162738Z:'` NOTE: The key point is how x-ms-ratelimit-remaining-resource has two key-value pairs: Microsoft.Compute/GetVMScaleSet3Min;197 - I ran this command before, so I have 197 requests available in the 3 minute window for performing GET requests on the VMSS resource Microsoft.Compute/GetVMScaleSet30Min;1297 - I now have 1297 requests available in the 30 minute window for performing GET requests on VMSS resources NOTE: x-ms-ratelimit-remaining-subscription-reads doesn’t seem to decrease (11999). Even if I run the same command again. I haven’t figured that out yet. Designing with quotas and throttling in mind Most Azure deployments won’t need this type of fine tuning, but just in case, there’s some documented Throttling Best Practices as well as my personal pro-tips: Use the Azure SDK, as many services have the recommended retry guidance built-in Instead of creating and deleting VMSS (which consume multiple VMSS API requests), scale the VMSS to 0 (which only consumes 1 VMSS API request) Any type of Kubernetes cluster auto-scaler will perform a reconciliation loop with Azure Compute RP. This could eat into your throttling limits Use the Azure Quota Service API to programmatically request quota increases If you’re unable to workaround the throttling limits, then the next step is to look at the Deployment Stamp pattern using multiple subscriptions. You can programmatically create subscriptions using Subscription vending. Hopefully this article has helped you understand quotas limits and throttling limits in Azure, and how to work around them. Let me know if you have any additional questions and/or feedback and I can follow-up with additional details.

engineering azure devops
· 5 min read

Creating a Dev Env with Azure Functions for Python

Azure Functions (one of the many Serverless Platforms inside Azure) allows you to use Python as your runtime environment. This is great; however, it requires Python 3.6, which is a problem for my development on a Mac (which uses Python 3.7). Downgrading/dual installation has the potential for many perils, so I wanted an isolated runtime development environment which wouldn't interfere with my current setup. Here's my requirements: Run Azure Functions locally (e.g. "func host start") Publish Azure Functions locally (e.g. "func azure functionapp publish") Use my existing Azure CLI session (i.e. don't have to login each time) Won't confuse my existing Python 3.7 env. Docker to the rescue! I'll spare you the details of the iterative process of creating the Dockerfile, but after a some iterations, I knew I was on the right track. You can copy/create my code here: https://gist.github.com/lastcoolnameleft/05b6b09735fb435b2cb4469f6cf30ac6 In short, it creates a Docker image and runs it with: Ubuntu 18.04 Python 3.6 Azure CLI Azure Function Core Tools Forwards port 7071 from inside the container to my localhost (used for local function testing) Mounts my home dir to /root to maintain my Azure CLI session. (e.g. No login!) This will definitely save me time each time I want to setup a new Function (or other) environment and I hope it helps save time for you too. Make my func the p-func!

engineering azure devops
· 2 min read

How I learned to stop worrying and love containers

I was recently invited to participate in the Microsoft Partner blog where I shared my love of containers. I'm especially passionate about container technology because of how much it makes the developer's life easier. Unfortunately, it's one of those things that must be experienced to truly understand. I tried to boil my thoughts town to just a few paragraphs here. Check it out and let me know what you think! https://blogs.technet.microsoft.com/msuspartner/2017/11/13/how-i-learned-to-stop-worrying-and-love-the-containers/

engineering containers devops
· 1 min read

LetsEncrypt on Azure App Service for Linux

Azure App Service for Linux is a pretty neat offering from Azure. You get all of the DevOps features you want (A/B Testing, Hosted Application, Tiered Support, Button-click scaling, lots of templates and more!) without the headache of managing VM's. 9 years ago, I wrote a quacky little website called "Duckiehunt". Unfortunately, I didn't pay the tech debt and things kept breaking until it was abandoned. I'm now using Duckiehunt as a learning ground for Azure's services and alternatives. Azure App Service for Linux was the perfect fit. However, back in 2008 SSL wasn't as ubiquitous. Now, it's a badge of shame to NOT have it. Azure does offer an App Service Certificate, but I'd like to find a cheaper/more open solution. Enter Let'sEncrypt from Mozilla and the EFF. If you don't know, EFF are the unsung heroes of the internet. They fight tirelessly to support your freedom and rights on the internet. Mozilla and EFF offer Let'sEncrypt as a free way to encrypt websites via CertBot. Now I'll dig into the technical details behind encrypting an App Service for Linux with Let'sEncrypt. Step #1: Get CertBot Because I'm on OSX, I was able to run: brew install certbot. For the full range of options, CertBot's webpage has what you need. Step #2: Create Cert locally Before CertBot can create the certificate for you, it must first validate you own the domain. It will prompt you for a few questions, and then ask you to create a file on the webhost and add content to that file for validation. Thankfully, Azure App Service for Linux provides a terminal access to your container so you can make these modifications yourself. ➜ sudo certbot certonly -d duckiehunt.com --manual ... Create a file containing just this data: %RANDOM STRING 1% And make it available on your web server at this URL: http://duckiehunt.com/.well-known/acme-challenge/%RANDOM STRING 2% ------------------------------------------------------------------------------- Press Enter to Continue Step #3: Add the validation file to you website I then went to the Kudu instance of my App Service and ran: ➜ mkdir /var/www/html/.well-known/acme-challenge/ ➜ echo "%RANDOM STRING 2%" > %RANDOM STRING 1% At this point, the validation is in place and it's time to continue with Chatbot by pressing "Enter". Waiting for verification... Cleaning up challenges IMPORTANT NOTES: - Congratulations! Your certificate and chain have been saved at: /etc/letsencrypt/live/duckiehunt.com/fullchain.pem Your key file has been saved at: /etc/letsencrypt/live/duckiehunt.com/privkey.pem Your cert will expire on 2017-11-12. To obtain a new or tweaked version of this certificate in the future, simply run certbot again. To non-interactively renew all of your certificates, run "certbot renew" - If you like Certbot, please consider supporting our work by: Donating to ISRG / Let's Encrypt: https://letsencrypt.org/donate Donating to EFF: https://eff.org/donate-le Huzzah! I've now got a certificate. Time to upload. Step #4: Upload the certificate to Azure Azure has a pretty descriptive set of steps for associating a certificate to your App Service, which I was able to follow. Openssl will ask for a Password which you need to keep as you upload the cert to Azure. ➜ cd /etc/letsencrypt/live/duckiehunt.com ➜ openssl pkcs12 -export -out myserver.pfx -inkey privkey.pem -in fullchain.pem Enter Export Password: Verifying - Enter Export Password: ➜ cp myserver.pfx ~/Desktop Step #5: Bind the certificate to your App Service From here on you're ready to Bind your SSL Certificate to your App Service. I'll let Microsoft's documentation lead the way from here. https://docs.microsoft.com/en-us/azure/app-service-web/app-service-web-tutorial-custom-ssl#bind-your-ssl-certificate Step #6: Bask in doing your part to secure the internet. In summary, the process was pretty painless. I used Let'sEncrypt to create a new Certificate for my App Service for Linux by creating a file that Let'sEncrypt could use to validate I owned the site. I then encrypted that certificate to upload to Azure. Once it was uploaded, I bound that certificate to my domain and voila! A more secure Duckiehunt One bummer is that the certificate is intended to expire in 3 months instead of the industry standard of 12 months. The renewal process looks pretty easy, but that's a different blog post. --Tommy feels that he's done his part in making the world a bit safer.

engineering azure devops
· 4 min read