DevCon24 Workshop Post-Mortem

This whole thing deserves its own blogpost, it was such a situation I won't include the long narration in the main blogpost of DevCon for fear of boring my poor readers.

What Happened?

This segment is for all of you who didn't attend the workshop in the first place. The idea was to have attendees each have their own cluster, provisioned from a "master cluster" through the magic of vcluster. I created a platform called Hobby Raft to dynamically provision vclusters, and retrieve the kubeconfig files for attendees to authenticate with them.

My issue on the day itself could be traced to several issues. Namely invalid config files and difficulties installing the kubectl command line utility to interact with the cluster.

How It Started

The root of all issues was storage. The days leading up to the workshop, I decided to pre-create 32 clusters. This would have avoided attendees waiting for their cluster to be spun up in the live workshop. The problem is that I was using Longhorn for storage, and it automatically replicates storage across, by default, 3 replicas. Each cluster needed 5GB for persistent storage, although in retrospect, this number could have been lower.

This meant that the first 12 clusters spun up fine. However, the remaining 20 were unable to as there was insufficient storage. After reducing the replicas from 3 to 1, and deleting the extra replicas, there was enough storage for the remaining clusters to start up.

Little did I know however, that the API I had made had timed out and these clusters had incomplete kubeconfig files.

Icing on the Cake

I had brought my old OpenSUSE laptop to act as ssh server. So people who couldn't install kubectl on their own devices could ssh to my laptop (ssh is a built-in Windows tool) and use kubectl there. However, I had forgotten to install kubectl for the `devcon` user I had created, and repository errors prevented me from doing so :(

Oh, and another thing. The grav cms image I had been using for the tests wasn't on docker hub anymore. In retrospect, I should've definitely tested that grav image or built my own that morning haha.

Conclusion

In conclusion, this semi-disaster was entirely due to my own mistakes of firstly not checking the kubeconfig files, and then not checking that essential tools were installed on the "ssh server". Thankfully, the feedback received was rather positive, taking that into account and I was happy to engage in a lot of fun discussions about Kubernetes. After changing the workshop into a prolonged demo, and shaking off the stress, it went pretty alright, despite some points where I was faced with the typical Kubernetes challenges (indentation) :P

A special mention from the bottom of my heart to Girish and Neil who were by my side throughout the 2h long workshop. Their support really gave me courage through the whole thing.