A gateway cluster is a set of virtual machines running gateway software that function as a single logical machine. The gateway cluster provides:

  • High availability against the failure of a node in the cluster.
  • High availability against the failure of a physical server on which nodes run.
  • Flexible horizontal scaling of nodes to manage more IT assets.

How a gateway cluster works

A gateway cluster is a set of virtual machines (VMs), which run applications that discover and monitor your environment. Gateway nodes run on physical servers that run a hypervisor, typically, with other VMs unrelated to the gateway. Nodes use a shared NFS storage volume to persist states shared among gateway nodes.

Gateway Cluster

Gateway Cluster

Each node also runs a lightweight Kubernetes - MicroK8s distribution. Kubernetes enables gateway nodes to work as a single, logical machine, which automatically schedules gateway applications between nodes. If a node fails or the host on which the node runs fails, or both a node and a host fail restarts applications on a different node, the logical node restarts the applications. The following figure illustrates how a gateway cluster works:

Gateway Cluster

Gateway Cluster: failure and recovery

Deployment options

Gateway clusters can be deployed in several configurations, depending on availability and horizontal scaling goals. The following figures illustrate three design points:

Gateway Cluster

Gateway Cluster

Gateway Cluster

Gateway Cluster

Prerequisites

To deploy a gateway cluster, make sure your environment meets these requirements:

Prerequisites
ComponentAttributeRequirements
NodesSize4 CPU cores, 8 GB RAM, and 40 GB Disk
NodesIP addressesStatic IP address allocation requirement:
  • One static IP address assigned to the primary network interface of each node.
  • One additional floating IP (static IP) address for the Gateway Manager application.
  • None of these IP addresses should be allocated to a DHCP pool.
  • All IP addresses must be in the same subnet.
NodesHostnameEach node should have a unique hostname. IP address and hostname should be added in **/etc/hosts** file and all nodes should be pingable with IP address and hostname.
NodesNetwork accessAll VMs should have outbound internet access to `*.opsramp.com` for connectivity, `k8s.gcr.io` and Google Artifact registry `us-docker.pkg.dev` to download gateway applications.
NodesNumber of nodesMinimum of three nodes for high availability.
HostsOSVMWare vSphere ESXi v6.0 or later versions.
HostsNumber of hostsMinimum of three hosts, running one gateway node on each host for high availability.
StorageTypeNFS storage volume, with read/write access from all nodes.

Set up a multi-node cluster

To set up a gateway cluster:

  1. Spin up nodes.
  2. Install MicroK8s
  3. Join nodes to the cluster
  4. Deploy gateway services on nodes.
  5. Register the Cluster Gateway.

The following description assumes a 3-node gateway cluster with each node running on a separate host.

See the troubleshooting section if you encounter problems.

Spin up nodes

The gateway is available as an Open Virtual Machine Appliance (OVA).

  1. Download the OVA.

  2. Spin up identical VMs on separate hosts, from the OVA, one VM for each node of the cluster.

  3. Log in to each VM with the default credentials provided to you.

    • Make sure to change the password.
    • You now have a VM for each node of the cluster.
  4. Set the new unique hostname on each node

    
        > sudo hostnamectl set-hostname <name>
        
  5. Add the IP address and hostname of each node in /etc/hosts file

    
        > sudo nano /etc/hosts
        

Install MicroK8s

  1. Run the following commands in all the nodes

    
        > cd /var/lib/node-manager/
        > sudo python main.py --install
        
  2. Check the MicroK8s status in all the nodes, it should be in the running state.

    
        > sudo microk8s status
        

Join nodes to the cluster

With three gateway nodes running, join them to create a cluster.

Select one of the nodes as your first node in the cluster, called node 1. You do not need to take an action to add the first node to the cluster.

Join node 2

To add the second node, called node 2, to the cluster. Go to the node 1 and run


> microk8s add-node

This returns joining instructions:


> microk8s join ip-172-31-20-243:25000/DDOkUupkmaBezNnMheTBqFYHLWINGDbf

If the node you are adding is not reachable through the default interface, use:


> microk8s join 10.1.84.0:25000/DDOkUupkmaBezNnMheTBqFYHLWINGDbf
> microk8s join 10.22.254.77:25000/DDOkUupkmaBezNnMheTBqFYHLWINGDbf

Copy the above command and run it on node 2.


> microk8s join ip-172-31-20-243:25000/DDOkUupkmaBezNnMheTBqFYHLWINGDbf

Wait for the process to complete on node 2.

Join node 3

To join the third node, called node 3, to the cluster, repeat the same steps for joining node 2.

To check that the node are successfully added, run:


> microk8s kubectl get nodes

Deploy gateway services on nodes

With the cluster set up, deploy gateway applications to the cluster.

Enable DNS

Run the following commands on any node to enable DNS


> microk8s enable dns

Install Storage, MetalLB and Gateway manager

Run the following commands on any of the nodes to install Persistent storage, MetalLB, and Gateway manager services.

  1. Create a folder to download the files

  > mkdir /var/lib/node-manager/charts
  > cd /var/lib/node-manager/charts
  
  1. Pull the required services

  > helm chart pull us-docker.pkg.dev/opsramp-registry/gateway-cluster-charts/gateway-extras:1.0.0
  > helm chart pull us-docker.pkg.dev/opsramp-registry/gateway-cluster-charts/kubernetes-dashboard:0.9.6
  > helm chart pull us-docker.pkg.dev/opsramp-registry/gateway-cluster-charts/gateway-manager:0.9.8
  
  1. Export the requried services

  > helm chart export us-docker.pkg.dev/opsramp-registry/gateway-cluster-charts/gateway-extras:1.0.0
  > helm chart export us-docker.pkg.dev/opsramp-registry/gateway-cluster-charts/kubernetes-dashboard:0.9.6
  > helm chart export us-docker.pkg.dev/opsramp-registry/gateway-cluster-charts/gateway-manager:0.9.8
  
  1. Configure the Storage type, NFS IP Address, Path, MetalLB IP

  > nano /var/lib/node-manager/charts/gateway-extras/values.yaml
  
  1. Provide the storage type as nfs, nfs IP Address and path and MetalLB IP as shown below

  storageType: &storageType nfs
  nfsServerIp: &nfsServerIp 172.25.251.89
  nfsServerPath: &nfsServerPath /srv/nfsdemo
  metallbIp: &metallbIp 172.25.252.100/32  #Use ip/32 format
  
  1. Install Storage, MetalLB and Gateway manager

  > helm install gateway-extras /var/lib/node-manager/charts/gateway-extras
  > helm install kubernetes-dashboard /var/lib/node-manager/charts/kubernetes-dashboard --namespace kubernetes-dashboard --create-namespace --debug
  > helm install gateway-manager /var/lib/node-manager/charts/gateway-manager -f gateway-manager/cluster.yaml --set secrets.defaultPassword=<Pass@1234> --debug
  
  1. Log in to the Gateway manager to register the gateway.
  • Open a web browser
  • Access http://<IPAddress>:5480

Note: IP assigned to MetalLB should be used.

Register Cluster Gateway

Log in to Gateway Manager to register the cluster:

  1. Go to Setup > Management Profiles.
  2. Create a new management profile and copy the activation token.
  3. Enter the activation token into the Gateway Manager.

Wait for the cluster registration to complete. You should see the status of the management profile turn to connected in the UI.

Migrate from a classic gateway

You can migrate an existing non-clustered, classic gateway to a cluster gateway without re-onboarding the resources managed by the classic gateway. To migrate from classic to clustered gateway:

  1. Create a new gateway cluster. Follow the steps to set up a multi-node cluster.
  2. Select the management profile associated with the classic gateway.
  3. De-register the classic gateway from the management profile.
  4. Register the new gateway cluster with the same management profile.

Existing configurations in the classic gateway are automatically migrated to the gateway cluster. Discovery and monitoring resume automatically after a few minutes.

Upgrading gateway firmware

To upgrade the gateway firmware:

  1. Swap out the existing VM.
  2. Deploy a new VM with the latest OVA.
  3. Attach the new VM to the same management profile.

FAQs

Can I roll back to the classic gateway from a gateway cluster?

  • Yes, you can roll back from a gateway cluster to the classic gateway without re-onboarding your managed resources and without loss of monitoring data.
  • De-register the gateway cluster from its management profile and register a classic gateway in its place.

How do I validate the multi-node cluster gateway setup?

Validate that your cluster is set up correctly by:

  • Running network discovery on a few managed resources.
  • Applying ping monitors on the resources.

How do I validate that the cluster can successfully recover from a node failure?

  1. Select one of the nodes.
  2. Power the node off to simulate node failure. The cluster recovers from the failure by restarting monitoring applications running on the failed node on the remaining nodes.
  3. Verify that monitors resume monitoring by observing the metric graphs in the UI.

Troubleshooting

If you can not ping the other nodes with the hostname, add the IP address and hostname of other nodes in the /etc/hosts file. See the MicroK8s documentation for tips on troubleshooting cluster setup-related issues.