Minikube on a VPS from scratch

It’s not rocket science to set up a Kubernetes (k8s) cluster on your local machine with minikube. On the other hand, It’s not straightforward when you want to set up minikube on a remote server. So, in this tutorial, I will share some learning after setting up my personal development environment in a VPS.

Configure access to the remote machine

By default, you will be provided with root credentials to access the machine. I hardly encourage you to perform the following 2 steps (and the 3rd is up to you)

1. Add your user account to the Server. Visit the following link if you need some help https://www.cyberciti.biz/faq/add-new-user-account-with-admin-access-on-linux/
2. Enable ssh key-based authentication for the user created in the previous step. It’s a properer access mechanism than using a user password. https://www.digitalocean.com/community/tutorials/how-to-configure-ssh-key-based-authentication-on-a-linux-server
3. Create a subdomain (or a domain) for your remote machine. It’s always easier to recall a name than an IP with 12 digits. We just need to configure a custom DNS record of type A. See below my own configuration:

• Hostname: k8s.ivancorrales.com
• Type: A
• TTL: 1h
• Data: 173.249.xxx.xxx (The public IP of your server)

Once you have completed the above steps you should be able to login into the remote machine as below:

ssh <user>@subdomain.domain (-i path_to_private_key)
# You don't need to pass the arg -i when using 
# the default key (~/.ssh/id_rsa)

# ssh icorrales@k8s.ivancorrales.com - 

Install minikube on the remote machine

Follow the installation guide https://minikube.sigs.k8s.io/docs/start/

Let’s create our first cluster as shown below [replace k8s.ivancorrales.com with the name of your own server]

minikube start --memory 8192 --cpus 4  \
  --insecure-registry="k8s.ivancorrales.com:5000"

We will launch a private docker registry to push our docker images, which will be deployed onto our k8s cluster. That’s the reason why we pass the flag --insecure-registry

Let’s install the addon registry

minikube addons enable registry

We will use the registry addons to set up our private docker registry that will host the docker images that will be deployed in our k8s cluster.

Once minikube is running we can check the status of our cluster with minikube status.

minikube status

minikube
type: Control Plane
host: Running
kubelet: Running
apiserver: Running
kubeconfig: Configured

Install kubectl on both remote machines & local machine

Follow the instructions at https://kubernetes.io/docs/tasks/tools/

SSH Tunneling

To minimize risks and avoid security risks we try to expose a less number of ports to the rest of the world. Therefore, we will use our ssh connection to create a tunnel to communicate with our k8s cluster that is running on the remote machine. From a simplistic point of view, we can sum up that a tunnel allows us to establish the connections that we could do from the remote machine.

If we run minikube with the docker driver (default behavior) we can run the command docker ps to verify the ports that are exposed from the container.

127.0.0.1:49212->22/tcp, 
127.0.0.1:49211->2376/tcp, 
127.0.0.1:49210->5000/tcp, 
127.0.0.1:49209->8443/tcp,  # API Endpoint
127.0.0.1:49208->32443/tcp

To figure the cluster IP out we just need to run the following:

>> minikube ip
192.168.49.2

In this tutorial, we will exclusively focus on the port 8443 , the one that is used by the tools to communicate with a k8s cluster via API.

So we need to work out how to communicate with the private IP 192.168.49.2 on port 8443 from our local machine.

The following command creates a tunnel that enables us to access ports 8443 ( k8s API ) through port 18443 in our local machine

ssh -N -p 22 icorrales@k8s.ivancorrales.com \
 -L 18443:192.168.49.2:8443 # k8s cluster

If we open https://localhost:18443 on our browser we will see a forbidden error message (Don’t panic, that error makes sense since this communication must be established using the certificates that we will download from the remote server in the following step.

The following image represents how the communication is established from our local machine to the k8s cluster through the tunnel

Set the k8s cluster connection

In our local machine, we need to edit the config file in path ~/.kube/config

• First of all, we need to download the minikube cluster certificates from the remote machines to our local machine. We can use thescp command to copy the certificates from the server to our local machine. [Replace the values with tour our username and server name]

mkdir -p ~/k8s.ivancorrales.com
cd ~/k8s.ivancorrales.com
scp icorrales@k8s.ivancorrales.com:/home/icorrales/.minikube/ca.crt .
scp icorrales@k8s.ivancorrales.com:/home/icorrales/.minikube/profiles/minikube/client.crt .
scp icorrales@k8s.ivancorrales.com:/home/icorrales/.minikube/profiles/minikube/client.key .

• Secondly, we need to add the k8s configuration in the file ~/.kube/config. This file is used to manage different k8s clustered, so you could have already defined others. Let’s see only the

# This file continas 3 blocks: cluster , contexts and user..
apiVersion: v1
clusters:
 - cluster:
      # Path to the ca.crt downloaded previously from the server
      certificate-authority: /Users/icorrales/k8s.ivancorrales.com/ca.crt
      # Port open in out local machine thorugh the tunnel
      server: https://127.0.0.1:18443
    name: k8s.ivancorrales.com
contexts:
  - context:
      # The value of the property cluster must match 
      # one of the defined users in the above block clusters
      cluster: k8s.ivancorrales.com
      namespace: default
      # The value of the property user must match 
      # one of the defined users in the below block users
      user: admin@k8s.ivancorrales.com
    name: minikube@k8s.ivancorrales.com
# Default context
current-context: minikube@k8s.ivancorrales.com
kind: Config
preferences: {}
users:
  - name: admin@k8s.ivancorrales.com
    user:
      # Path to the client.crt downloaded previously from the server
      client-certificate: /Users/icorrales/k8s.ivancorrales.com/client.crt
      # Path to the client.key downloaded previously from the server
      client-key:  /Users/icorrales/k8s.ivancorrales.com/client.key

Assuming that we have defined the value of. current-context with the new context that we have defined, we can verify that everything is correctly configured using the kubectl tool that we previously installed on our machine.

>> kubectl cluster-info
Kubernetes control plane is running at https://127.0.0.1:18443
CoreDNS is running at https://127.0.0.1:18443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

Admin the cluster remotely

Now that we have correctly created the communication we can manage our k8s cluster from our local machine.

For instance, we can create a namespace in our cluster

>> kubectl create namespace demo
namespace/demo created
>> kubectl delete namespace demo
namespace "demo" deleted

Dashboard

To launch the k8s dashboard we need to run the below command from the remote machine

# From remote machine
minikube dashboard

and launch a proxy from the local machine

# local machine
kubectl proxy --address='0.0.0.0' --disable-filter=true
....
Starting to serve on [::]:8001
....

The address http://localhost:8001/ displays the different endpoints that we navigate to

{
  "paths": [
    "/.well-known/openid-configuration",
    "/api",
    "/api/v1",
    "/apis",
    .....
   ]
}

The dashboard can be accessed by clicking on the link http://localhost:8001/api/v1/namespaces/kubernetes-dashboard/services/http:kubernetes-dashboard:/proxy/

Publish a docker image

Build a Docker image

At the beginning of this article, we enabled the addon registry

Let’s tag any docker image that we want to display in the k8s cluster. Alternatively, you could use the docker ivancorrales/hello-world:tutorial that launch an HTTP server on port 3030 that displays hello world! when you access to /

We need to tag the image to be hosted in our Docker registry.

docker tag ivancorrales/hello-world:tutorial \
  k8s.ivancorrales.com:5000/hello-world:0.0.1

Check the docker registry logs

The docker registry is a docker container that runs in the namespace kube-system . We can check the logs as shown below

kubectl logs --namespace kube-system service/registry -f

Make the Docker registry accessible from your local machine

Run the following command from the remote machine

kubectl port-forward --namespace kube-system service/registry 5000:80 \
  --address 0.0.0.0

Let’s open the following URL (don’t forget to replace k8s.ivancorrales.com with your own subdomain) http://k8s.ivancorrales.com:5000/v2/_catalog and we can check the response from the registry

{"repositories":[]}

Before pushing any docker image we need to add the URL k8s.ivancorrales.com:5000 as an insecure registry in our docker configuration.

{
  
  "insecure-registries": [
    "k8s.ivancorrales.com:5000"
  ]
}

Publish the docker image

We can publish the Docker image that we created previously.

docker push k8s.ivancorrales.com:5000/hello-world:0.0.1

Install our service with Helm

To complete this tutorial we will use Helm to deploy an application.

Install Helm in tour local machine

Follow the instructions here, https://helm.sh/docs/intro/install/

Write your Helm Chart

Learning to write a Helm Chart is out of the scope of this tutorial, so you can clone this repository and replace the values in the variables with the ones that work for you. The Chart is under folder hello-world

Install the application

If you decided to clone the repository, from the root directory you just need to run

helm install my-hello-worl-app hello-world --namespace minikube-demo \
  --create-namespace --debug

and check that the chart was installed correctly.

>> helm  list --namespace minikube-demo
...
>> kubectl logs deployment/myapp-deployment --namespace minikube-demo 
> docker-hello-world@1.0.0 start
> node index.js

Listening on port 3030

Check the service

We just need to port-forwarding the deployment port from our local machine

kubectl port-forward deployment/myapp-deployment \
  --namespace minikube-demo 3030:3030

So let’s open http://localhost:3030 to check our beautiful hello world!