This article explains why, when, and how to use self-managed Kubernetes clusters in Azure for testing custom scenarios.

Kubernetes has gotten so large and complex that most companies prefer to use the managed service (e.g. AKS, GKE) instead of running it themselves. By using a managed Kubernetes service, this frees up the operations team to focus on their core competency instead of optimizing, backing up and upgrading of Kubernetes.

While this reduces the operational burden, you lose the ability to modify the platform. Sometimes these are acceptable tradeoffs, sometimes you need to manage it yourself.

Historically, AKS-engine was the OSS tool for creating unmanaged Kubernetes clusters on Azure, but it had some limitations. CAPI/CAPZ is the go-forward solution for creating and operating self-managed clusters declaratively.

I highly recommend reading Scott Lowe’s article on An introduction to CAPI. It covers a lot of terminology and concepts used here.

One of the reasons for using CAPI/CAPZ is as a testing and development tool for Kubernetes on Azure. For example, you might need to build and test the following scenarios:

• A kernel change to the worker nodes
• A modification to the K8S config on control plane nodes
• An installation of a different CNI
• The use of K8S to manage K8S

This diagram represents a high level architecture of a starter CAPI/CAPZ cluster.

The rest of this article will explain how to implement the above scenarios utilizing the CAPI quickstart. Because the command arguments will change over time, this article will describe the steps and provide a link to the full details like this:

Link to CAPI Quick Start with details: base command to run

Create the KIND Cluster

Similar to RepRap, CAPI uses a Kubernetes cluster to make more Kubernetes clusters. The easiest way is with Kuberenetes IN Docker (KIND). As the name implies, it’s a Kubernetes cluster which runs as a Docker container. This is our starting point for what we call “Bootstrap Cluster”.

Create Kind Cluster: kind create cluster

Initialize cluster for Azure

We will use this bootstrap cluster to initialize the “Management Cluster” which contains all of the CRDs and runs the CAPI controllers. This is where we will apply all of our changes to meet our scenarios.

Initialize cluster for Azure: clusterctl init --infrastructure azure

Generate cluster configuration

Now that our management cluster is ready, we want to define what our workload cluster will look like. Thankfully, there are different flavors we can pick from. By using the default, we will get an unmanaged K8S cluster using virtual machines.

Generate cluster configuration: clusterctl generate cluster capi-quickstart > capi-quickstart.yaml

We now have a file which contains the CRDs which will define our workload cluster. We will modify capi-quickstart.yaml and edit the CRDs to implement each of our scenarios.

Full documentation is available for CAPI (baseline) CRDs and CAPZ (Azure specific resources) CRDs.

Scenario: Worker node kernel change

If we want to modify the worker nodes, we likely want to add a preKubeadmCommands and postKubeadmCommands directive in the KubeadmConfigTemplate.

preKubeadmCommands allows a list of commands to run on the worker node BEFORE joining the cluster.

postKubeadmCommands allows a list of commands to run on the worker node AFTER joining the cluster.

apiVersion: bootstrap.cluster.x-k8s.io/v1alpha4
kind: KubeadmConfigTemplate
metadata:
  name: capi-quickstart-md-0
  namespace: default
spec:
  template:
    spec:
      preKubeadmCommands:
        - wget -P /tmp https://kernel.ubuntu.com/<path>.deb
        - dpkg -i /tmp/<package name>.deb
      postKubeadmCommands:
        - reboot

After you’ve made these changes, you can proceed to the rest of the steps by applying the resources to your management cluster which will then create your workload cluster and deploy the CNI.

Scenario: Modify Kubernetes components

If we want to modify the control plane, we can make changes to the KubeadmControlPlane. This allows us to leverage the kubeadm API to customize various components.

For example, to enable a Feature Gate on the kube-apiserver:

apiVersion: controlplane.cluster.x-k8s.io/v1alpha4
kind: KubeadmControlPlane
metadata:
  name: capi-quickstart-control-plane
  namespace: default
spec:
  kubeadmConfigSpec:
    clusterConfiguration:
      apiServer:
        extraArgs:
          feature-gates: MyFeatureGate=true

The above example omits some fields for brevity. Make sure that you keep any existing args and configurations that you are not modifying in-place.

After you’ve made these changes, you can proceed to the rest of the steps by applying the resources to your management cluster which will then create your workload cluster and deploy the CNI.

Apply the Workload Cluster

Now that we have defined what our cluster should look like, apply the resources to the management cluster. The CAPZ operator will detect the updated resources and talk to Azure Resource Manager.

Apply the workload cluster kubectl apply -f capi-quickstart.yaml

Monitor the Cluster Creation

After you’ve made the changes to the capi-quickstart.yaml resources and applied them, you’re ready to watch the cluster come up.

Watch the cluster creation:

• kubectl get cluster
• clusterctl describe cluster capi-quickstart
• kubectl get kubeadmcontrolplane – Verify the Control Plane is up

Now that the workload cluster is up and running, it’s time to start using it!

Get the Kubeconfig for the Workload Cluster

Now that we’re dealing with two clusters (management cluster in Docker and workload cluster in Azure), we now have two kubeconfig files. For ease, we will save it to the local directory.

Get the Kubeconfig for the workload cluster clusterctl get kubeconfig capi-quickstart > capi-quickstart.kubeconfig

Install the CNI

By default, the workload cluster will not have a CNI and one must be installed.

Deploy the CNI kubectl --kubeconfig=./capi-quickstart.kubeconfig apply -f https://...calico.yaml

Scenario: Install a different CNI

If you want to use flannel as your CNI, then you can apply the resources to your management cluster which will then create your workload cluster.

However, instead of Deploying the CNI, you can follow the steps in the Install Flannel walkthrough.

Cleanup

When you’re done, you can cleanup both the workload and management cluster easily.

Delete the workload cluster kubectl delete cluster capi-quickstart

If you want to create the workload cluster again, you can do so by re-applying capi-quickstart.yaml

Delete the management cluster kind delete cluster

If you want to create the management cluster again, you must start from scratch. If you delete the management cluster without deleting the workload cluster, then the workload cluster and Azure resources will remain.

Summary

Similar to how Kubernetes allows you to orchestrate containers using a declarative syntax, CAPI/CAPZ allows you to do the same, but for Kubernetes clusters in Azure.

This article covered example scenarios for when to use CAPI/CAPZ as well as a walkthrough on how to implement them.

I’m especially excited for the future of CAPI/CAPZ and how it can integrate with other Cloud Native methodologies like GitOps to declaratively manage clusters.

P.S. I am extremely grateful to Cecile Robert Michon’s (Twitter & Github) technical guidance for this article. Without her support, I wouldn’t have gotten this far and definitely would have missed a few key scenarios. Thanks Cecile!

I scoffed when I heard wearable technology was an upcoming “big thing” for fashion. However, since basking in the warm glowing, warming glow of my first LED on an Arduino, I was hooked.

When my buddy, Dan Stach, started showcasing his breakthroughs in lanyard technology, I knew he was onto something. And I knew I wanted to turn it to 11.

I got tired of boring lanyards, so I made my own. pic.twitter.com/LPd8Roi6rd

— Tommy Falgout (@lastcoolname) June 5, 2019

We iterated over and over until we streamlined the design to be portable enough to fit behind the badge. After rigorous field-testing (i.e.: lot of tech conferences), the design now fits in a small travel case and can be assembled in minutes.

If you are interested in building your own, you should know a few things about the build and procurement process:

Straps

We tried a few different straps and the black one from Strapworks looks the coolest. The white strap looks ok, but the black one makes the design look very 8-bit.

Controller case

This has the most potential for customization and fun. My original case was custom co-designed with the owner of Bricks and Minifigs, Dallas (a used LEGO retailer). I’ve streamlined this to use LEGO + tack to make the box smaller.

For a while I considered selling LED lanyards, and planned to use a 4″x6″ craft case, but was unsatisfied with the results. Every option has felt too bulky or awkward for my taste while LEGO hits that perfect intersection of cool, functional and modular.

LED Controller

The controller is what you use to turn the LED on & off as well as display patterns. The cheapest/easiest is to purchase an RF LED controller because it just works and comes with a remote for ~$11.

I really wanted to go take it to the next level so I used a RaspberryPi Zero W for extra customization. This presented NUMEROUS challenges which is deserving of a separate post. If you are interested in pursuing this, contact me to discuss.

Can’t believe the nerd quotient of my good friend Tommy — @lastcoolname. Ever heard of a lanyard that you could SSH into or that changes colors using bot framework? Yeah, me neither. Until now. Wow. #iot #RaspberryPi @OpenAtMicrosoft pic.twitter.com/5qfnyzzLWm

— Stuart R. Kirk – Microsoft 🐧 (@stuartatmsft) July 24, 2019

LED strip

There are many LED options, but the ones we wanted were WS2812B, which used RGB LED, could be individually addressed, and would easily talk with our LED controller. We also liked the 144 pixels per meter which gave it a nice density. We also liked IP30 which mean it had no plastic coating on top. This made it more likely to break, but more flexible and would dissipate the heat better.

Battery

Come to find out, managing electricity is hard. We lost many hours debugging why the LED would power off when changing patterns. Cheap wires were the culprit as the controller couldn’t pull enough current for some configurations (e.g. all white LED). A credit-card battery worked best because of the bendable power cord which was useful in the tight space of the controller box and minimized our wire problems.

Summary

If by the end of this post, you’re still interested in making your own, you’re in luck! I created a parts list and assembly instructions.
Please let me know how it goes (share your pics!) and if there’s anything I can do to improve the experience.

— Tommy wants to be the light of your life. Well, at least your lanyard.

Azure Functions (one of the many Serverless Platforms inside Azure) allows you to use Python as your runtime environment.

This is great; however, it requires Python 3.6, which is a problem for my development on a Mac (which uses Python 3.7). Downgrading/dual installation has the potential for many perils, so I wanted an isolated runtime development environment which wouldn’t interfere with my current setup.

Here’s my requirements:

• Run Azure Functions locally (e.g. “func host start”)
• Publish Azure Functions locally (e.g. “func azure functionapp publish”)
• Use my existing Azure CLI session (i.e. don’t have to login each time)
• Won’t confuse my existing Python 3.7 env.

Docker to the rescue!

I’ll spare you the details of the iterative process of creating the Dockerfile, but after a some iterations, I knew I was on the right track.

You can copy/create my code here:
https://gist.github.com/lastcoolnameleft/05b6b09735fb435b2cb4469f6cf30ac6

In short, it creates a Docker image and runs it with:

• Ubuntu 18.04
• Python 3.6
• Azure CLI
• Azure Function Core Tools
• Forwards port 7071 from inside the container to my localhost (used for local function testing)
• Mounts my home dir to /root to maintain my Azure CLI session. (e.g. No login!)

This will definitely save me time each time I want to setup a new Function (or other) environment and I hope it helps save time for you too.

Make my func the p-func!

The internet’s a weird place. We already knew that, yet it keeps finding new ways to amaze me.

Someone thought it would be a novel idea to incorporate Alexa with a novelty electronic fish (Big Mouth Billy Bass). Now you can ask a fish for the current weather and the fish can tell you if it’s a good day to catch its brethren.

I take that back. The world is a weird place. And I embrace it and want to leave my mark on it.

So, when given an opportunity to build something similar, a robotic version of Clippy was the only natural solution. People have re-fallen in love with Clippy.

Ya girl got business cards 😎📎☁️ pic.twitter.com/1nNBYvlZ1t

— Chloe Condon 🎀 (@ChloeCondon) February 21, 2019

However, one thing was missing. A proper, physical manifestation of our favorite sentient paperclip. This is the story of that journey.

RoboClippy Mark I

RoboClippy Mark I was cute, but definitively not a paperclip. I could get the eyebrows to wiggle, but only manually.

RoboClippy Mark II

Upon creating RoboClippy Mark II, I realized that the concept of “Uncanny Valley” applies to both humans and paperclips.

RoboClippy Mark III

In RoboClippy Mark III, I realized that animatronics done wrong is nightmare fuel.

But I made progress! The eyebrows articulate (not realistically), but it doesn’t look natural or have fine motor control.

Since I want to use motors to control LEGO, the LEGO Mindstorms sounded perfect! Alas, I encountered a number of issues interacting with their LEGO power supplies.

• LEGO Mindstorm motors use 9V; however, I wanted to power everything from USB which only uses 5V
• LEGO Mindstorm connectors have weird wiring and I wanted more standard cabling

There’s a whole field of hobby electronics and motors, so I went down that path. I soon learned that not all motors are created equal. Servo motors are good for broad movements where you apply voltage to tell the motor “turn left” or “turn right”. Stepper motors are good for fine movements where you apply voltage to tell the motor “Go to position X”.

After some experimentation, stepper motors became the natural fit. The catch is that they use a special protocol to control them called PWM. And if you want to control multiple motors, you want to use a different protocol (I2C). In my case, I wanted to control 3 motors (mouth, left eyebrow, right eyebrow). This was getting more complex than I expected, but I was learning a lot and excited.

RoboClippy Mark IV

RoboClippy Mark IV was a technological breakthrough. With the help of my good friends at Bricks and Minifigs, Plano we had a working prototype which looked realistic and could articulate it’s mouth.

Remember Uncanny Valley? Without the eyebrows, it looks … off.

So, how are we doing so far? We’ve got a great structure, the mouth articulates, and the eyebrows articulate! However, it’s lacking a “soul”. We want it to move the mouth when speaking, we want it to simulate Clippy’s voice and we want to use the eyebrows to emote.

Enter Azure Cognitive Services. There are many services it offers, and in this case I’ll be using Azure Speech to Text and Text to Speech so that I can give RoboClippy a voice and to listen to what people are saying.

Now for the next problem: Determining when RoboClippy is speaking. It seems intuitive to have RoboClippy’s mouth move when the audio is playing and stop when it’s complete, right? Uncanny Valley wins again. If you see someone’s mouth moving when they’re not speaking (e.g. at the end of a sentence), it doesn’t look right.

So, the next option is to calculate voltage off of the soundcard/speaker, right? Again, there’s more nuances to be discovered. Sound is a wave, so measuring at any point only gets you a snapshot. Also most microphones measure -2.5V to 2.5V and the Arduino can only detect 0 – 5V, so we’re missing half the data! A Step-up Converter fixed that problem, but added additional complexity.

RoboClippy Mark V

Now we’re at RoboClippy Mark V. I’m using an Arduino to measure the sound and control the servos. All of the communication happens on my laptop. It’s powered and controlled by the USB. Unfortunately, the results were really flakey and since it drew power from the laptop, there was a potential of the motors drawing too much current and frying it. It also took about 5-10 minutes to setup and get right each time. Major progress! But not very practical.

Inspired by the Big Mouth Billy Bass, I built a @LEGO #Clippy #robot using @Azure Huge thanks to @geekpatrol and Bricks and Minifigs, Plano pic.twitter.com/Fay08GUs4I

— Tommy Falgout (@lastcoolname) October 1, 2018

My local Makerspace had some Google AIY Voice Kits which I experimented with. It had a speaker, a microphone, and a cardboard case, all you needed was to supply your own Raspberry Pi. This was exactly the packaging I needed to contain RoboClippy’s brains.

The last step is making RoboClippy “talk”. But some interesting questions arose:

• How do I know when to start listening? Wait for a user to press a button? Not a great experience.
• How do I know when to stop listening? Again, not a great experience.
• What’s the quickest way to respond? Perform S2T & T2S locally? Use a service?
• How can I best utilize Azure? This is Clippy, so using MS products makes sense.

Thankfully, someone wrote an OSS library to solve many of these problems. I also learned something about Alexa/Cortana/OK Google that I wasn’t aware of. Keyword detection (aka Hotwork detection).

Anyone with an Amazon Dot/Echo is worried that Alexa/Amazon is always listening in on us. Keyword detection is training an AI model to “wake-up” and do something when it hears specific pitches/frequencies. You can even create you own keyword! Enter Snowboy, a service from Kitt.AI for making your own keyword. This allows RoboClippy to wake-up upon that specific pitch/frequency and then start “really” listening on the microphone. Thankfully, the same OSS library supported Snowboy so this was surprisingly easy to incorporate. You can even contribute to the “Hello Clippy” keyword.

RoboClippy Mark VI

Now, witness the power of this fully armed and operational RoboClippy.

• Our RoboClippy is now MUCH more extensible.
• Google hardware for microphone and speakers
• Microsoft Azure for Text2Speech, Speech2Text, Natural Language Processing
• I2C to PWM for motor controls
• RaspberryPi for orchestration
• Power + control (ssh + Python) can be done remotely
• 5 easy-to-connect wires (4 for I2C, 1 USB for power)
• Written in Python
• Available as Open Source

Build your own Robo-Clippy

To build your own, you will need:

• Raspberry Pi 3B+ – You might be able to use an older version, but I’m not sure how Snowboy will work since it can be resource intensive.
• LEGO – Exact parts TBD, but mostly: black 1-width bricks; black plates; grey 2-width bricks, grey plates.
• Azure Cognitive Services – T2S/S2T – You can sign up for a free account on Azure.
• Google AIY Voice Kit
• I2C PWM Driver – This might not be essential as the Google AIY Voice Kit allows you to connect multiple servos directly to the HAT. By the time I got to this stage, I had already integrated the board and I prefer to keep it this way because it makes cabling easier.
• Multi-color breadboard jumper wires – These are used for connecting the RPi HAT to the Servos.
• 3 Micro Servo Motor
• Snowboy – Hotword/keyword detection
• Python 3 – Glue code

In an upcoming blog, I will detail the steps necessary to create your own. If you can’t wait and want to start working on one now, feel free to email me at tommy at this domain.

If you’re interested in seeing my presentation on this story, you can view it here:

Special thanks:

• Special thanks to Jason and Andrea of Bricks and Minifigs, Plano who helped design LEGO Clippy
• Greg Miller who helped me understand the properties of sound via oscilloscope
• Nina Zakharenko who was the catalyst for this blog post
• Chloe Condon who is an even bigger Clippy fan than myself and helped fuel this social rebirth

As much as Cloud Providers tout their availability and uptime, disasters happen. It’s inevitable. And it’s usually up to you to be prepared. There are services that can help; however, they’re not always “Kubernetes aware”.

Thankfully, the great folks at Heptio open-sourced Ark, a Disaster Recovery tool which works for all the major cloud providers.

I got hands-on with Ark and followed their Azure steps. It was a good start, but didn’t highlight how an actual failover and recovery would look to the operator. I took their steps and created a step-by-step guide to perform a full migration.

Ark support Azure native resources, namely Managed Disk + Snapshots.
You can review those steps here: https://github.com/heptio/ark/blob/master/docs/azure-config.md

Another option would be to use Restic, which performs backups to a local file system. Later, I’ll detail the steps on how to use Restic with Azure.

If you’re looking for Best Practices on supporting Business Continuity and Disaster Recovery for AKS/K8S clusters in Azure, you’re in luck! I wrote a Microsoft article covering this use case, which can be found here:
https://docs.microsoft.com/en-us/azure/aks/operator-best-practices-multi-region

I created this article with the intent of explaining the migration journey from deploying a legacy application with manual steps to an automated Kubernetes deployment with proper DevOps practices. Its intent is not to help you understand Kubernetes deeper (there’s an abundance of materials out there already).

As a Cloud Solution Architect for Microsoft, every week I work with our partners to assist them towards containerization and Kubernetes. I’ll use AKS and discuss it’s strengths and weaknesses without holding punches. Disclaimer: Given I work for Microsoft, I am self-aware of my bias. So in this article, I will make an effort to be more critical of Azure to balance that out.

Beginning With the End in Mind, I created the following outline:

Intent

Duckiehunt is secure, monitored and deployable with the least amount of manual effort, cost and code-change.

Purpose

I wrote Duckiehunt in 2007 as a LAMP website. It embodies many of the customer requirements I see:

• Old code, using legacy tooling
• Want a reliable, resilient infrastructure
• Want to automate deployment
• Don’t want to re-write
• Migration should involve minimal/no code change
• Need to update to modern standards (e.g. HTTPS, MySQL encryption, private DB instance with backups)

Outcomes

• CI/CD (Code Check-in triggers automated tests and pushes to Production)
• Monitoring cluster + app (visualization + alerts if down)
• HTTPS enabled for duckiehunt.com (CA Cert + forced redirection to https)
• Running on Kubernetes (AKS)
• Managed MySQL

Milestones: (in reverse order of accomplishment)

• Production DNS migrated
• Azure Monitor + Container Monitoring Solution + LogAnalytics
• Distinct Dev + Prod environments
• VSTS + Github integration
• Securely expose UI + API
• Integrated MySQL instance
• Installed on AKS
• Test in Minikube
• Migrate App to Container

From here on, I’ll explain my journey as steps fulfilling the milestones I created. I’ll list my estimated time, as along with my actual time to compare. The times below are not “Time to get X working”, but “Time to get X working correctly and automate as if I had to support this in production” (which I do). As a result, they’re much higher than a simple success case.

Migrate app to Container

Estimated Time: 4 hours. Actual Time: 10 hours

I wrote this in 2007 using a PHP version that is no longer supported (5.3) and a framework (CodeIgniter) that is not as active. I didn’t want to re-write it yet. Thankfully, 5.6 is mostly backwards compatible and I was able to find a container using that.

I would have been done in ~4 hours; however, I lost an embarrassing amount of hours banging my head against the wall when I automated the docker build. (I would always get 404) I learned this was because Linux’s file system is case-sensitive and OSX’s is not, and the PHP framework I chose in 2007 expects the first character of some files to start with a capital letter. *grumble* *grumble*

Test in Minikube

Estimated time: 12 hours. Actual Time: 10 hours

Now that I got my PHP app running in a container, it was time to get it running inside Kubernetes. To do this, I needed to deploy, integrate and test the following: Pod, Service, Secrets, Configuration, MySQL and environment variables.

This is a pretty iterative approach of “This, this…nope…how about this?…Nope…This?…ah ha!…Ok, now this…Nope.” This is where Draft comes in. It’s a Kubernetes tool specifically designed for this use case, and I think I’ve started to develop romantic feelings for this tool because of how much time and headache it saved me while being dead simple to use.

Install in AKS

Estimated time: 8 hours. Actual time: 2 hours

Creating a new AKS cluster takes about 10 minutes and is instantly ready to use. Because I had done the work on testing it Minikube the hard-word was already done, but I expected some additional hiccups. Again, this is where my love and adoration of Draft started to shine. I was almost done in 30 minutes, but I took some shortcuts with Minikube that came back to bite me.

Integrated MySQL instance

Estimated time: 2 hours. Actual time: 3 hours

Azure now offers MySQL as a Service (aka Azure Database for MySQL) and I chose to use that. I could have run MySQL in a container in the cluster; however, I would have had to manage my own SLA, backups, scaling, etc. Given my intent of this project is to have the least amount of work and cost, and the cost is still within my MSDN budget, I chose to splurge.

I spent an hour experimenting with Open Service Broker for Azure (a way of managing external dependencies, like MySQL, native to K8S). I really like the idea, but I wanted one instance for both Dev + Prod and needed a high control over how my app read in database parameters (since it was written in 2007). If I was doing more deployments than one, OSBA would be the right fit, but not this time.

Steps taken:

1. Create the Azure Database for MySQL Instance
2. Created the dev/prod accounts
3. Migrated the data (mysqldump)
4. White-listed the source IPs (To MySQL, the cluster traffic looks as if it’s coming from the Ingress IP address)
5. Injected the connection string to my application (Using K8S Secrets)

Then I was off to the races. OSBA would have automated all of that for me, but I’ll save that for a proverbial rainy day.

Securely expose UI + API

Estimated time: 4 hours. Actual time: 20 hours

This was the most frustrating part of the entire journey. I decided to use Nginx Ingress Controller with Cert-manager (for SSL). There’s lots of old documentation that conflicts with recommended practices, which led to lots of confusion and frustration. I got so frustrated I purposely deleted the entire cluster and started from scratch.

Lessons’ learned:

1. nginx-ingress is pretty straight-forward and stable. Cert-manager is complicated and I had to restart it a lot. I really miss kube-lego (same functionality, but deprecated. Kube-lego was simple and reliable)
2. Put your nginx-ingress + cert-manager in kube-system, not in the same namespace as your app
3. You might have to restart cert manager pods when you modify services. I had issues where cert-manager was not registering my changes.
4. cert-manager might take ~30 minutes to re-calibrate itself and successfully pull the cert it’s been failing on for the last 6 hours
5. cert-manager creates secrets when it tries to negotiate, so be mindful of extra resources left around, even if you delete the helm chart
6. cert-manager injects its own ingress into your service for verifying you own the domain. If you don’t have your service/ingress working properly, cert-manager will not work
7. If you’re doing DNS changes, cert-manager will take a long time to “uncache” the result. Rebooting kibe-dns doesn’t help.
8. There’s no documentation for best-practices for setting up 2 different domains with cert-manager (e.g. dev.duckiehunt.com; www.duckiehunt.com)
9. AKS’s HTTP application routing is a neat idea, but you cannot use custom domains. So you’re forced to use its *.aksapps.io domain for your services. Great idea, but not useful in real-world scenarios

To summarize, I was finally able to get development and production running in two different namespaces with one ingress controller and one cert-manager. Should have been simple, but death-by-1000-papercuts ensued with managing certs for each of them. Now I’m wiser, but the journey was long and frustrating. That might involve a blog post of its own.

VSTS + Github integration

Estimated time: 4 hours. Actual time: 2 hours

VSTS makes CI/CD easy. Real easy. Almost too easy.

I lost some time (and ~8 failed builds) because the VSTS UX isn’t intuitive to me and documentation is sparse. But now that it’s working, I have a fully automated Github commit -> Production release pipeline which completes within 5 minutes. This will save me a tremendous amount of time in the future. This is what I’m most excited about.

Azure Monitor + Container Monitoring Solution + LogAnalytics

Estimated time: 3 hour. Actual time: None.

This was the surprising part. All of this work was already done for me by setting up the AKS cluster and integrated into the portal. I was impressed that this was glued together without any additional effort needed.

That said, here’s some “gotchas”:

• The LogAnalytics SLA is ~6 hours. My testing showed that new logs showed up within 5 minutes, but after a cluster is newly created, initial logs would take ~30 minutes to appear.
• The LogAnalytics UX isn’t intuitive, but the query language is extremely powerful and each of the pods logs were available by clicking through the dashboard.
• Monitoring and Logging are two pillars of the solution; however, Alerting is missing from the documentation. That integration is forthcoming, and will likely involve another blog entry.
• The “Health” tile is useful for getting an overview of your cluster; however, the “Metrics” tile seems pretty limited. Both are still in Preview, and I expect to see additional improvements coming soon.

Production DNS migrated

Estimated time: 1 hour. Actual time: 1 hour

Since I did the heavy lifting in the “Securely expose UI + API” section, this was as easy as flipping a light switch and updating the DNS record in my registrar (dreamhost.com). No real magic here.

Summary

This has been a wonderful learning experience for me, because I was not just trying to showcase AKS/K8S and its potential, but also using it as it is intended to be used, thus getting my hands dirtier than normal. Most of the underestimated time was spent on a few issues that “rat-holed” me due to technical misunderstandings and gaps in my knowledge. I’ve filled in many of those gaps now and hope that it saves you some time too.

If this has been valuable for you, please let me know by commenting below. And if you’re interesting in getting a DuckieHunt duck, let me know as I’d love to see more take flight!

P.S. The source code for this project is also available here.

WARNING: SSH’ing into an agent node is an anti-pattern and should be avoided. However, we don’t live in an ideal world, and sometimes we have to do the needful.

Overview

This walkthrough creates an SSH Server running as a Pod in your Kubernetes cluster and uses it as a jumpbox to the agent nodes. It is designed for users managing a Kubernetes cluster who cannot readily SSH to into their agent nodes (e.g. AKS) does not publicly expose the agent nodes for security considerations).

This is one of the steps in the Kubernetes Workshop I have built when working with our partners.

NOTE

It has been tested in AKS cluster; however, it should also work in other cloud providers.

You can follow the steps on the SSH to AKS Cluster Nodes walkthrough; however, that requires you to upload your Private SSH key which I would rather avoid.

Assumptions

* The SSH Public key has been installed for your user on the Agent host
* You have jq installed Not vital, but makes the last step easier to understand.

Install an SSH Server

If you’re paranoid, you can generate your own SSH server container; however, [this one by Corbin Uselton](https://github.com/corbinu/ssh-server) has some pretty good security defaults and is available on Docker Hub.

kubectl run ssh-server --image=corbinu/ssh-server --port=22 --restart=Never

Setup port forward

Instead of exposing a service with an IP+Port, we’ll take the easy way and use kubectl to port-forward to your localhost.

NOTE: Run this in a separate window since it will need to be running for as long as you want the SSH connection

kubectl port-forward ssh-server 2222:22

Inject your Public SSH key

Since we’re using the ssh-server as a jumphost, we need to inject our SSH key into the SSH Server. Using root for simplicity’s sake, but I recommend a more secure approach going forward. (TODO: Change this to use a non-privileged user.)

cat ~/.ssh/id_rsa.pub | kubectl exec -i ssh-server -- /bin/bash -c "cat >> /root/.ssh/authorized_keys"

SSH to the proxied port

Using the SSH Server as a jumphost (via port-forward proxy), ssh into the IP address of the desired host.

# Get the list of Host + IP's
kubectl get nodes -o json | jq '.items[].status.addresses[].address'
# $USER = Username on the agent host
# $IP = IP of the agent host
ssh -J root@127.0.0.1:2222 $USER@$IP

NOTE: If you get “WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!” You might need to add `-o StrictHostKeyChecking=no` to the SSH command if you bounce across clusters. This is because SSH believes that the identity of the host has changed and you need to either remove that entry from your `~/.ssh/known_hosts` or tell it to ignore the host identity.

Cleanup

• kubectl delete pod ssh-server
• Kill the kubectl port-forward command

This week, I found myself in one of the most unique and challenging situations of my life. And now that it’s all over, I find myself in tears. Not because of sadness, but because I now know myself as someone who can actually make a difference is this world, despite the circumstances.

Now for a little backstory.

It should be no surprise that I love to build. I found my best friend, Lee Gibson, when a LEGO set came up at a White Elephant party and we both schemed on how to win it. I’ve created a non-profit called “The Trebuchet Society”, with the primary goal of hosting SlingFest, a (mostly) annual event designed to gather builders from around the area to create trebuchets and toss pumpkins hundreds of feet. It’s a blast and fuels my desire to build and be around other builders.

In 2014, I discovered TheLab.ms via a tweet. A budding Makerspace/Hackerspace. Its mission is to foster a collaborative environment wherein people can explore and create intersections between technology, science, art, and culture.

I found my people.

Their guiding principles were more focused on education and ethical hacking instead of building trebuchets, but that’s cool. My mom was a librarian, so education is in my blood. I just wanted to be around like-minded people.

I watched Shawn Porter, Roxy Dehart and Richard Gowen pour their heart out into it and build it from scratch. TheLab even got an article in the Plano Magazine.

As with all non-profits, you want awareness, engagement and members. These usually bring in new ideas and fresh blood. Sometimes in alignment with your own ideas, sometimes not. And as a father, I can tell you, there is no rage in the world like watching something happen to your baby.

Fast forward a few years and after some leadership changes, the last of the founders resigned as a board member, and a number of positions were either vacant or MIA. Then the Education Coordinator resigned. Then the President resigned. Then the Floating Board Member. And the Vice President. And the Secretary.

Their reasons were their own. And I support them 100%.

I was now in one of the most unique and confronted situations of my life. The sole Board Member of TheLab.ms. A community that I’ve been with from almost the very start and loved so dearly was fighting amongst itself. Anger and frustration was evident on a daily basis. People were burnt out.

Thankfully, I had an ace in my pocket. For the last 6 months, I’ve been registered in a course called “Team Management and Leadership Program” from Landmark Worldwide. It is a course designed around creating teams and teamwork in any situation that produce powerful results in many areas of life with freedom and ease. I called my coach and the classroom leader in tears that day. I felt completely broken down and had no idea how to make this work. Through an insightful and “tough love” conversation, I started to see a path forward.

I organized a last-minute event and invited people to create the future of TheLab. I expected about 6 people to show up. I had to hold back my emotions when the room completely filled up, including members I hadn’t seen in years. These were people who, despite the burn-out, despite the anger, despite the frustration, deeply wanted TheLab to not just survive, but to thrive. It was showtime.

In an hour and a half, we dug deep, asked some good questions and had some fun. We had some deep, meaningful conversations about the future and not the past. And most importantly, people stepped up to the plate to take on some big leadership positions. Elections are next week and I invite all of you to learn what we’re about. I have never been more proud to be part of an organization than I am right now.

I have found my people.

Again.

Emerging civilizations naturally gravitate towards beds of water. Growing up in lower Louisiana, the Mighty Mississippi was where my ancestry settled. It was a source of commerce, livelihood and fisheries which provided sustainability that allowed the surrounding areas to flourish to the ecosystem it is now.

Technology mimics this cultural expansion and KubeCon/CloudNativeCon is the riverbed where developers and operators around the world arrive to ship and receive containers from the Kubernetes dock.

I was fortunate enough to join 50+ other Microsoft’ers and 4000+ others KubeCon/CloudNativeCon on Dec 5-8th. This hotbed of activity has flourished from the internal foundational work that Google created to a vibrant open source community. This small stream has gathered enough momentum to be undeniable in the development and operations community.

Kubernetes is software that makes it easier to run your software. Software development is hard, not just because you have to worry about your code, but you also have to worry about monitoring, maintaining, updating, scaling and more. Kubernetes was the pilot program for a larger organization called the Cloud Native Compute Foundation. The CNCF was designed to be stewards for this and other projects with the intention of making software easier to develop and operate.

If you missed the event, and want to vicariously live through my notes, you’re in luck as I keep pretty detailed notes:
https://github.com/lastcoolnameleft/Conference-Notes/tree/master/KubeCon-2017

This year was the year of the Service Mesh and socks.

The week was not just an opportunity to learn from other experts, but to be at the forefront of new announcements from my favorite cloud.

Azure Announcements:

• Virtual Kubelet – The new version of the Kubernetes connector was announced at KubeCon. This enables Azure to extend Kubernetes to Azure Container Instances (ACI), and provide our customers with per-second billing and NO virtual machine management for containers.
• Ark – a migration tool which enables teams to move AWS and GCP (cross cloud Kubernetes tool) to Azure. Microsoft and Heptio (the creators of Ark) have formed a strong partnership. Ark delivers a strong Kubernetes disaster recovery solution for customers who want to use it on Azure.
• Open Service Broker for Azure – We announced the open sourcing the Open Service Broker for Azure (OSBA), built using the Open Service Broker API. OSBA exposes popular Azure services to Kubernetes such as Azure CosmosDB, Azure Database for PostgreSQL, and Azure Blob Storage.
• Metaparticle – Brendan Burns announced during the Keynote address, the delivery of an experimental model for coding for cloud. Metaparticle attempts to reduce the complexity and duplication of code for deploying software to Kubernetes.
• Kashti – A visualization dashboard for https://github.com/azure/brigade

Other notable announcements:

• Kubeflow – Machine Learning Toolkit for Kubernetes
• Alibaba Cloud is a platinum member of CNCF
• Codefresh announces support for Helm Charts in Kubernetes
• CoreOS Tectonic 1.8 released
• Oracle announces new open source Kubernetes Tools
• Weaveworks Cloud Enterprise Edition
• Many more that I’ve forgotten or didn’t jot down

Oh, and it snowed in Austin. It was a KubeCon Miracle!

P.S. A special shout-out to my travel/seminar buddies, Al Wolchesky, Kevin Hillinger, Nick Eberts, Brian Redmond and Eddie Villalba.

I was recently invited to participate in the Microsoft Partner blog where I shared my love of containers.

I’m especially passionate about container technology because of how much it makes the developer’s life easier. Unfortunately, it’s one of those things that must be experienced to truly understand. I tried to boil my thoughts town to just a few paragraphs here. Check it out and let me know what you think!

https://blogs.technet.microsoft.com/msuspartner/2017/11/13/how-i-learned-to-stop-worrying-and-love-the-containers/