vcluster: How to Create Virtual Kubernetes Clusters

Publié le 28/11/2023

⭠ Back to the list of articles

Here is a technical English translation of the document:

1. vcluster: How to Create Virtual Kubernetes Clusters Within a Real Cluster

Second article in a series on "multi-tenant" topics.

Kubernetes is a container orchestration system that allows you to deploy, manage, and scale distributed applications. Kubernetes provides a high-level abstraction for manipulating resources like pods, services, deployments, etc. These resources are organized into namespaces, which are logical grouping and isolation units.

However, namespaces are not enough to guarantee complete isolation between different users or environments within the same Kubernetes cluster. Indeed, there are resources that are global to the cluster, like Custom Resource Definitions (CRDs), Cluster Roles, Persistent Volumes, etc. These resources can conflict with or be shared between namespaces, causing security, performance, and compatibility issues.

To solve these problems, there is a solution that consists of creating virtual Kubernetes clusters, which are full-fledged Kubernetes clusters that run within a namespace of a real Kubernetes cluster. These virtual clusters have their own API server, their own control plane, and their own resources. They are therefore more powerful and more isolated than regular namespaces.

2. What is vcluster?

vcluster is an open source tool that makes it easy to create virtual Kubernetes clusters. vcluster is based on k3s, a lightweight Kubernetes distribution developed by Rancher. vcluster uses a component called syncer, which synchronizes low-level resources of the virtual cluster, like pods, services, ingresses, with the real cluster. Thus, the real cluster runs these resources, but the virtual cluster controls them.

vcluster has several advantages over other Kubernetes multi-tenancy solutions:

It is compatible with any Kubernetes cluster, without having to install agents or operators on the real cluster.
It is easy to use, just one command or API call to create a virtual cluster. It is cost-effective, it saves resources by sharing the real cluster's control plane and worker nodes.
It offers better isolation and resilience than namespaces, avoiding conflicts or failures related to global cluster-wide resources.

3. How to Use vcluster

3.1 Using the CLI

To use vcluster, you first need to install the vcluster CLI on your local machine.

The easiest way is to follow the procedure described here: (https://www.vcluster.com/docs/getting-started/setup)

The vcluster CLI allows you to create, delete, list and connect to virtual clusters. You also need access to a real Kubernetes cluster, which will serve as the host for the virtual clusters.

To create a virtual cluster, you can use the following commands:

# Create a new vCluster in the vCluster-my-vcluster namespace

vcluster create my-vcluster

# OR: use --expose to create a vCluster in a remote cluster with a LoadBalancer accessible from outside
vcluster create my-vcluster --expose

# OR: use -f to use an additional helmvalues.yaml with additional chart options to deploy the vCluster
vcluster create my-vcluster -f values.yaml

# OR: use --distro to specify k0s or vanilla k8s as the virtual cluster underlay
vcluster create my-vcluster --distro k8s

# OR: use --isolate to create an isolated environment for vCluster workloads
vcluster create my-vcluster --isolate

Example:

❯ vcluster create vcluster-herlec-k8s  --distro k8s -n herlec-k8s --expose
11:15:00 info Creating namespace herlec-k8s
11:15:00 info Create vcluster vcluster-herlec-k8s...
11:15:00 info execute command: helm upgrade vcluster-herlec-k8s /var/folders/h2/12qsmrms09b5d8_g44gl87zm0000gn/T/vcluster-k8s-0.17.1.tgz-978776111 --kubeconfig /var/folders/h2/12qsmrms09b5d8_g44gl87zm0000gn/T/521084887 --namespace herlec-k8s --install --repository-config='' --values /var/folders/h2/12qsmrms09b5d8_g44gl87zm0000gn/T/2392899474
11:15:03 done Successfully created virtual cluster vcluster-herlec-k8s in namespace herlec-k8s
11:15:03 info Waiting for vcluster to come up...
11:16:03 info Using vcluster vcluster-herlec-k8s load balancer endpoint: 4.225.40.250
11:16:03 done Switched active kube context to vcluster_vcluster-herlec-k8s_herlec-k8s_k8s-ims-admin
- Use `vcluster disconnect` to return to your previous kube context
- Use `kubectl get namespaces` to access the vcluster

❯ kubectl get namespaces
NAME              STATUS   AGE
default           Active   46s
kube-node-lease   Active   46s
kube-public       Active   46s
kube-system       Active   46s

❯ kubectl config get-contexts --output name

* k8s-ims-admin
vcluster_vcluster-herlec-k8s_herlec-k8s_k8s-ims-admin


❯ k get po -n herlec-k8s
NAME                                                           READY   STATUS    RESTARTS        AGE
coredns-68559449b6-hbqgf-x-kube-system-x-vcluster-herlec-k8s   1/1     Running   0               9m10s
vcluster-herlec-k8s-6cf689688f-xj9cx                           1/1     Running   0               10m
vcluster-herlec-k8s-api-6f784c8f97-zgnb7                       1/1     Running   1 (9m40s ago)   10m
vcluster-herlec-k8s-controller-788966cbdb-dhn9d                1/1     Running   2 (9m38s ago)   10m
vcluster-herlec-k8s-etcd-0                                     1/1     Running   0               10m

❯ kubectl config use-context vcluster_vcluster-herlec-k8s_herlec-k8s_k8s-ims-admin

❯ kubectl get po

No resources found in default namespace.

3.1.1 Common CLI Commands

To connect to the virtual cluster, you need to use the following command:

vcluster connect my-vcluster -n my-namespace

This command will open a port-forward to the virtual cluster and modify the KUBECONFIG environment variable to use the previously generated kubeconfig file. Thus, you can use kubectl or helm commands as if you were connected to the virtual cluster.

To list existing virtual clusters, you need to use the following command:

vcluster list

This command will display the name, namespace, status and URL of virtual clusters.

To pause a vCluster: To temporarily shrink the vCluster and remove all its workloads created on the host cluster. This can be useful to save resources used by vCluster workloads in the host cluster.

vcluster pause my-vcluster -n my-namespace

This command will:

Scale statefulset and deployment to '0'
Delete all workloads created by the vCluster
Resuming a vCluster

vcluster resume my-vcluster -n my-namespace
#or
vcluster connect my-vcluster -n my-namespace

To delete a virtual cluster, you need to use the following command:

vcluster delete my-vcluster -n my-namespace

3.2 Using Helm

Create the vcluster.yaml file:

vcluster:
  image: rancher/k3s:v1.23.5-k3s1

Then install the Helm chart using vcluster.yaml for chart values:

helm upgrade --install my-vcluster vcluster \
  --values vcluster.yaml \
  --repo https://charts.loft.sh \
  --namespace my-namespace \
  --repository-config=''

4. Advanced Usage

To set limits and quotas for clusters created with vcluster, you can use Kubernetes' ResourceQuota and LimitRange objects. These objects allow controlling the consumption and creation of cluster resources per namespace.

A ResourceQuota defines constraints that limit the total amount of resources a namespace can consume. For example, you can limit the number of pods, CPU, memory or persistent storage that a namespace can use. You can create a ResourceQuota in the namespace where the virtual cluster runs, using the vcluster CLI or applying a YAML file. Here is an example ResourceQuota that limits the namespace to 10 vCores, 20GB of memory and a maximum of 10 pods:

You just need to create a resource in the vcluster namespace

apiVersion: v1
kind: ResourceQuota
metadata:
  name: vcluster-quota
spec:
  hard:
    cpu: "10"
    memory: 20Gi
    pods: "10"

A LimitRange defines constraints that apply to resources requested or allocated by pods or containers in a namespace. For example, you can set default values or minimum/maximum values for CPU or memory resources of pods or containers. You can create a LimitRange in the namespace where the virtual cluster runs, using the vcluster CLI or applying a YAML file. Here is an example LimitRange that sets default values for container CPU and memory resources:

apiVersion: v1
kind: LimitRange
metadata:
  name: vcluster-limit-range
spec:
  limits:
    - default:
        memory: 512Mi
        cpu: "1"
      defaultRequest:
        memory: 128Mi
        cpu: 100m
      type: Container

To limit pod scheduling to selected nodes.

You need to follow the documentation available here: Limiting pod scheduling to selected nodes

5. What Are the Use Cases for vcluster?

vcluster can be used for several scenarios, such as:

Development and testing: you can create virtual clusters on the fly for each branch, each pull request or each test environment, without having to manage separate real clusters. This allows you to have isolated, reproducible and ephemeral environments, which are automatically destroyed at the end of the lifecycle.
Multi-tenancy: you can create virtual clusters for each user, team or customer, without having to share global cluster resources. This allows for better isolation, security and compatibility between tenants, who can use their own CRDs, cluster roles, etc.
Education and training: you can create virtual clusters for each student, intern or workshop participant, without having to provision real clusters for each one. This allows having 100% functional Kubernetes clusters, which can be used to learn or teach Kubernetes concepts and practices.

6. Conclusion

vcluster is an innovative and powerful tool that allows you to create virtual Kubernetes clusters within a real Kubernetes cluster. vcluster provides a Kubernetes multi-tenancy solution that is more powerful and isolated than regular namespaces. vcluster can be used for several use cases such as development, testing, multi-tenancy, education or training. vcluster is easy to use, just install the vcluster CLI and run a single command to create a virtual cluster.

If you want to learn more about vcluster, you can check out the following resources:

The vcluster website: vcluster.com
The vcluster GitHub repo: (https://github.com/loft-sh/vcluster)
The vcluster blog: vcluster.com/blog
The vcluster documentation: vcluster.com/docs
The vcluster introduction video at KubeCon 2023: (https://www.youtube.com/watch?v=FYqKQIthH6s)