Introduction

Linux cluster is a group of Linux computers or nodes, storage devices that work together and are managed as a single system. A traditional clustering configuration has two nodes that are connected to shared storage (typically a SAN). With Linux clustering, an application is run on one node, and clustering software is used to monitor its operation.

A Linux cluster provides faster processing speed, larger storage capacity, better data integrity, greater reliability and wider availability of resources.

Failover

Failover is a process. Whenever a primary system, network or a database fails or is abnormally terminated, then a Failover acts as a standby which helps resume these operations.

Failover Cluster

Failover cluster is a set of servers that work together to provide High Availability (HA) or Continuous availability (CA). As mentioned earlier, if one of the servers goes down another node in the cluster can take over its workload with minimal or no downtime. Some failover clusters use physical servers whereas others involve virtual machines (VMs).

CA clusters allow users to access and work on the services and applications without any incidence of timeouts (100% availability), in case of a server failure. HA clusters, on the other hand, may cause a short hiatus in the service, but system recovers automatically with minimum downtime and no data loss.

A cluster is a set of two or more nodes (servers) that transmit data for processing through cables or a dedicated secure network. Even load balancing, storage or concurrent/parallel processing is possible through other clustering technologies.

Linux Failover Cluster Monitoring

If you look at the above image, Node 1 and Node 2 have common shared storage. Whenever one node goes down, the other one will pick up from there. These two nodes have one virtual IP that all other clients connect to.

Let us take a look at the two failover clusters, namely High Availability Failover Clusters and Continuous Availability Failover Clusters.

High Availability Failover Clusters

In case of High AVailability Failover Clusters, a set of servers share data and resources in the system. All the nodes have access to the shared storage.

High availability clusters also include a monitoring connection that servers use to check the “heartbeat” or health of the other servers. At any time, at least one of the nodes in a cluster is active, while at least one is passive.

Continuous Availability Failover Clusters

This system consists of multiple systems that share a single copy of a computer’s operating system. Software commands issued by one system are also executed on the other systems. In case of a failover, the user can check critical data in a transaction.

There are a few Failover Cluster types like Linux Server Failover Cluster (WSFC), VMware Failover Clusters, SQL Server Failover Clusters, and Red Hat Linux Failover Clusters.

Pre-Requisites

  1. OpsRamp Classic Gateway 11.0 and above (or) OpsRamp Cluster gateway
  2. Ensure that “adapter integrations'' addon is enabled in client configuration. Once enabled you can see Linux Fail-over Cluster integration under Setup -> Integrations -> Adapter section.
  3. Credentials : root / non-root privileges with a member of “haclient” group.
  4. Cluster management: Pacemaker.
  5. Accessibility: All nodes within a cluster should be accessible by a single credential set.
  6. For non-root users: Update “~/.bashrc” file with “pcs” command path across all cluster nodes.
    Ex: export PATH=$PATH:/usr/sbin -> as a new line in ~/.bashrc file.

Install the integration

  1. From All Clients, select a client
  2. Go to Setup > Integrations > Integrations
  3. From Available Integrations, select Adapter > Linux Fail-over Cluster. Then Install Linux Fail-over Cluster Integration pop-up appears.
    Note: Ensure that Adapter addon is enabled at client and partner levels.
    Install Integration
  4. Enter the following information:
Object NameDescription
NameName of the Integration.
Upload LogoOptional logo for the integration.
GateWay ProfilesSelect a gateway management profile to associate with the clients.

To know more on how to create a GateWay Profile, refer to the register a gateway article.
5. Click Install. The Integration page displays the installed integration.
Linux Install Integration

Configure the integration

  1. In CONFIGURATION section, click +Add.

  2. On Create Adapter Configuration, enter

    • Name: Configuration name.
    • IP Address / Host Name: IP address or host name of the target.
    • Notification Alerts: Select TRUE or FALSE.
      Note: By default False is selected.
  3. From the Credentials section, select Custom, enter Username and Password.
    Note: These credentials are required to communicate with the target (cluster).

  4. From the Resource Types & Metrics section, select the metrics and configure for availability and alert conditions, for Cluster & Server.

  5. In the Discovery Schedule section, configure how frequently the discovery action should trigger. Select Recurrence Pattern to add one of the following patterns:

    • Minutes
    • Hourly
    • Daily
    • Weekly
    • Monthly
  6. In the Monitoring Schedule section, configure how frequently the monitoring action should trigger.

  7. Click Save.

    Linux Install Integration
    Linux Install Integration
    Linux Install Integration


    After saving the configuration, the resources are discovered and monitoring is done as specified in the configuration profile.
    The configuration is saved and displayed on the page.
    Linux Configuration

    You can perform the actions manually, like Discovery, Monitoring or even Disable the configuration.
    The discovered resource(s) are displayed in the Infrastructure page under “Cluster”, with Native Resource Type as Linux Cluster.
    Basic Information

    Note: The cluster nodes are displayed under Components.

Supported Metrics

Resource Type: Cluster

Metric NamesDescriptionDisplay NameUnit
linux_cluster_nodes_statusStatus of each node present in linux cluster.
0 - offline
1- online
2- standby
Cluster Node Status
linux_cluster_system_OS_UptimeTime elapsed since last reboot in minutes.System Uptimem
linux_cluster_system_cpu_LoadMonitors the system's last 1 min, 5 min and 15min load. It sends per cpu core load average.System CPU Load
linux_cluster_system_cpu_UtilizationThe percentage of elapsed time that the processor spends to execute a non-Idle thread(This doesn't includes CPU steal time)SSystem CPU Utilization%
linux_cluster_system_memory_UsedspacePhysical and virtual memory usage in GBSystem Memory Used SpaceGB
linux_cluster_system_memory_UtilizationPhysical and virtual memory usage in percentage.System Memory Utilization%
linux_cluster_system_memory_UtilizationMonitors cpu time in percentage spent in various program spaces
User - The processor time spent running user space processes.
System - The amount of time that the CPU spent running the kernel.
IOWait - The time the CPU spends idle while waiting for an I/O operation to complete. Idle - The time the processor spends idle.
Steal - Time the virtual CPU has spent waiting for the hypervisor to service another virtual CPU, running on a different virtual machine. Kernel Time Total Time
System CPU Usage Statistics%
linux_cluster_system_disk_UsedspaceMonitors disk used space in GBSystem Disk UsedSpaceGB
linux_cluster_system_disk_UtilizationMonitors disk utilization in percentageSystem Disk Utilization%
linux_cluster_system_disk_Inode_UtilizationThis monitor is to collect DISK Inode metrics for all physical disks in a server.System Disk Inode Utilization%
linux_cluster_system_disk_freespaceMonitors the Free Space usage in GBSystem FreeDisk UsageGB
linux_cluster_system_network_interface_Traffic_InMonitors In traffic of each interface for Linux DevicesSystem Network In TrafficKbps
linux_cluster_system_network_interface_Traffic_OutMonitors Out traffic of each interface for Linux Devices.System Network Out TrafficKbps
linux_cluster_system_network_interface_Packets_InMonitors In Packets of each interface for Linux Devices.System Network In packetspackets/sec
linux_cluster_system_network_interface_Packets_OutMonitors Out packets of each interface for Linux Devices.System Network out packetspackets/sec
linux_cluster_system_network_interface_Errors_InMonitors network in errors of each interface for Linux Devices.System Network In ErrorsErrors per Sec
linux_cluster_system_network_interface_Errors_OutMonitors Network Out traffic of each interface for Linux DevicesSystem Network Out ErrorsErrors per Sec
linux_cluster_system_network_interface_discards_InMonitors Network in discards of each interface for Linux Device.System Network In discardspsec
linux_cluster_system_network_interface_discards_OutMonitors network Out Discards of each interface for Linux Devices.System Network Out discardspsec
linux_node_system_OS_UptimeTime elapsed since last reboot in minutes.System Uptimem
linux_node_system_cpu_LoadMonitors the system's last 1 min, 5 min and 15min load. It sends per cpu core load average.System CPU Load
linux_node_system_cpu_UtilizationThe percentage of elapsed time that the processor spends to execute a non-Idle thread(This doesn't includes CPU steal time).System CPU Utilization%
linux_node_system_memory_UsedspacePhysical and virtual memory usage in GB.System Memory Used SpaceGb
linux_node_system_memory_UtilizationPhysical and virtual memory usage in percentage.System Memory Utilization%
linux_node_system_cpu_Usage_StatsMonitors cpu time in percentage spent in various program spaces
User - The processor time spent running user space processes.
System - The amount of time that the CPU spent running the kernel.
IOWait - The time the CPU spends idle while waiting for an I/O operation to complete.
Idle - The time the processor spends idle.
Steal - The time virtual CPU has spent waiting for the hypervisor to service another virtual CPU running on a different virtual machine. Kernel Time Total Time
System CPU Usage Statistics%
linux_node_system_disk_UsedspaceMonitors disk used space in GB.System Disk UsedSpaceGb
linux_node_system_disk_UtilizationMonitors disk utilization in percentage.System Disk Utilization%
linux_node_system_disk_Inode_UtilizationThis monitor is to collect DISK Inode metrics for all physical disks in a server.System Disk Inode Utilization%
linux_node_system_disk_freespaceMonitors the Free Space usage in GB.System FreeDisk Usage.Gb
linux_node_system_network_interface_Traffic_InMonitors In traffic of each interface for Linux Devices.System Network In Traffic.Kbps
linux_node_system_network_interface_Traffic_OutMonitors Out traffic of each interface for Linux Devices.System Network Out TrafficKbps
linux_node_system_network_interface_Packets_InMonitors in Packets of each interface for Linux DevicesSystem Network In packetspackets/sec
linux_node_system_network_interface_Packets_OutMonitors Out packets of each interface for Linux DevicesSystem Network out packetspackets/sec
linux_node_system_network_interface_Errors_InMonitors network in errors of each interface for Linux DevicesSystem Network In ErrorsErrors per Sec
linux_node_system_network_interface_Errors_OutMonitors Network Out traffic of each interface for Linux DevicesSystem Network Out ErrorsErrors per Sec
linux_node_system_network_interface_discards_InMonitors Network in discards of each interface for Linux DevicesSystem Network In discardspsec
linux_node_system_network_interface_discards_OutMonitors network Out Discards of each interface for Linux DevicesSystem Network Out discardspsec
linux_cluster_service_status_PacemakerPacemaker High Availability Cluster Manager. The status representation as follows :
0 - "failed", 1 - "active" & 2 - "unknown"
Pacemaker Service Status
linux_cluster_service_status_CorosyncThe Corosync Cluster Engine is a Group Communication System. The status representation as follows :
0 - "failed", 1 - "active" & 2 - "unknown"
Corosync Service Status
linux_cluster_service_status_PCSDPCS GUI and remote configuration interface. The status representation as follows :
0 - "failed", 1 - "active" & 2 - "unknown"
PCSD Service Status
linux_cluster_Online_Nodes_CountOnline cluster nodes countOnline Nodes Countcount
linux_cluster_Failover_StatusProvides the details about cluster failover status. The integer representation as follows:
0 - cluster is running on the same node, 1 - there is failover happened.
Cluster FailOver Statuscount
linux_cluster_node_HealthThis metrics gives the info about the percentage of online linux nodes available within a cluster.Cluster Node Health Percentage%