I'll preface this by saying that I don't know how to name things. So I'll just call this The Cluster for this blogpost. This project is basically a Kubernetes cluster which spans the entirety of the island of Mauritius. I gave a fun presentation on the v1 of this project at Rejekts in London earlier this year. You can watch the recording on the Rejekts YouTube channel. In this blogpost, I will reiterate a few of the principles behind the existence of this project and how I intend on creating v2.
Why Create This Cluster
For the uninitiated, Mauritius is a tiny island in the Indian Ocean, vaguely East of Madagascar. The French will undoubtedly know us as being next-door neighbours to La Reunion. The "why" behind the project is largely due to our geographical position.
- Cloud Service Providers are an ocean away. The nearest datacenters to Mauritius for CSPs such as Azure, Google Cloud, and AWS, are in South Africa. This means the latency is around 60ms, and bandwidth is limited, especially when there are pre-existing issues.
- Mauritius has no network redundancy. We are connected to the rest of the world through 3 undersea cables: METISS, LION2, and SAFE which have capacities of 24Tb, 1.3Tb, and 800Gb respectively. So, if Metiss has an issue, the capacity really available to Mauritius (and La Reunion) is some measly 2.1Tb (b = bits, not bytes). Not so long ago, there was damage on two individual cables which crippled critical digital infrastructure on the island as latency to Europe shot up to 800-1000ms and North America was practically inaccessible.
- Servers are expensive. First of all, we suffer extra because of shipping costs, which, I am convinced, account for 25% of the price of servers. Additionally, the only local CSP, cloud.mu tends to be a bit expensive, especially if you are not a business.
- Powercuts. While my homelab hosts a few "essential" services such as this website and meetup.mu, it's localised entirely in my living room and thus everything will go down if a powercut (which isn't rare) hits.
The Solution
I'll keep this part brief. The solution I came up with is to essentially treat the whole world as your homelab. So me and a few friends split up some micro PCs among ourselves, and linked them together through Tailscale which creates a virtual network through which the servers can communicate as in a mesh. I threw on K3s (for being lightweight) and Longhorn for storage. Minio is used as S3 backup target for Longhorn. Longhorn also takes care of replication for me, ensuring that volumes are replicated across servers in different houses, ensuring redundancy in case one "location" were to fail.
While the cluster had a few challenges, they have all since been fixed and the cluster is being [reliably] used by a local startup.
Version 2
I am quite proud of the v1 and how it overcame a lot of the unique challenges we face in Mauritius. However, I want to build a v2 soon with some important upgrades.
- More powerful nodes. The ones comprising v1 are all really old 3/4th gen i3/5 and, justifying the low cost, are extremely underpowered. They run hot and are inefficient as compared to more modern machines. And most have either 4 or 8GB of DDR3 RAM. This hurts me on a psychological level. v2 would hopefully feature 7th gen Intel CPUs and above, with DDR4 RAM.
I would nonetheless like to preserve the micro PC format as it is very small, quiet, and efficient, allowing people to run nodes without worrying about noise, heat, space, or electricity costs. - A larger community. V1 is spread across 3 people, each of whom has 3 nodes. I want v2 to involve more people and these people can choose how many nodes they want to host (no more than 3) depending on available space and bandwidth limitations.
- An open distributed infrastructure. As v1 is being used by a startup, it isn't exactly available for the general public to use. However, I have a concept of a plan for making v2 more open to the public. At the base, I want to use Vcluster or K3k to create virtual Kubernetes clusters residing within Kubernetes. People would then be given vclusters through an intuitive user interface, and provided a kubeconfig file in order to play around with Kubernetes and host their own projects on the virtual cluster. As it's virtual, people A and B would have vclusters on the same host cluster but be unable to interfere with each other's resources.
Initially I had thought of simply using RBAC for giving each person a namespace, but certain resources like PVs are cluster-wide and I don't want people messing with each other's data :) - Aaaand security. Right now I could theoretically spin up a ubuntu:latest pod and access other devices on the LAN of the host device. Though, now I think about it, would that work given it's on Tailscale... Either way, there's some additional research I need to do, particularly regarding NetworkPolicies. So that will be the biggest challenge.
Making It Happen
This project is significantly more ambitious than v1, and I personally lack the resources to make it happen. For starters, I need to find people willing to host servers as part of this project; thankfully I have a few in mind already. Second, I need to find money to acquire micro PCs (preferably Dell Optiplex Micros, or similar) or find a company which is getting rid of their out-of-warranty thin clients and which would be open to donating them. Admittedly, postage is still going to be annoying.
And while this is a community project, I would be thrilled if companies as well were to reach out to sponsor nodes or the project at large. So, now comes the part where I share my contact details to make v2 possible. If you would like to host a node at your house, or if you would like to sponsor the project through funding or server donations, I can easily be reached on Telegram or on mail@alexbissessur.dev !