Introduction

NVIDIA Bright Cluster Manager offers fast deployment, and end-to-end management for heterogeneous high-performance computing (HPC) and AI server clusters at the edge in the data center, and in multi/hybrid-cloud environments. It automates provisioning and administration for clusters ranging in size from a couple of nodes to hundreds of thousands. It also supports CPU-based and NVIDIA GPU-accelerated systems, and enables orchestration with Kubernetes.

Monitoring Use cases

  • In case of any threshold breach or unexpected metric behavior, based on configurations the device monitoring helps to collect the metric values with respect to time and sends alerts to the intended customer team to act up immediately.
  • It helps the customer with smooth functioning of business with minimal or zero downtime in case of any infrastructure related issues occurring.

Prerequisites

OpsRamp NextGen Gateway.

Default monitoring configurations

Bright Cluster Manager has default Global Device Management Policies, Global Templates, Global Monitors and Global Metrics in OpsRamp. You can customize these default monitoring configurations as per your business use cases by cloning respective Global Templates, and Global Device Management Policies. OpsRamp recommends doing this activity before installing the application to avoid noise alerts and data.

  1. Default Global Device Management Policies

    OpsRamp has a Global Device Management Policy for each Native Type of NVIDIA Bright Cluster Manager. You can find those Device Management Policies at Setup -> Resources -> Device Management Policies -> Search with suggested names in global scope. Each Device Management Policy follows below naming convention:

    {appName nativeType - version}

    Ex: NVIDIA BrightCluster Manager NVIDIA BCM Head Node (i.e, appName = nvidia-bright-cluster-manager, nativeType = NVIDIA BCM Head Node, version = 1)

  2. Default Global Templates

    OpsRamp has a Global Template for each Native Type of NVIDIA Bright Cluster Manager. You can find those templates at Setup -> Monitoring -> Templates -> Search with suggested names in global scope. Each template follows below naming convention:

    {appName nativeType 'Template' - version}

    Ex : nvidia-bright-cluster-manager NVIDIA BCM Head Node Template - 1 (i.e, appName = nvidia-bright-cluster-manager, nativeType = NVIDIA BCM Head Node Template, version = 1)

  3. Default Global Monitors

    OpsRamp has a Global Monitors for each Native Type which has monitoring support. You can find those monitors at Setup -> Monitoring -> Monitors -> Search with suggested names in global scope. Each Monitors follows below naming convention:

    {monitorKey appName nativeType - version}

    Ex: NVIDIA BCM Head Node Monitor nvidia-bright-cluster-manager NVIDIA BCM Head Node Container 1(i.e, monitorKey = NVIDIA BCM Head Node Monitor, appName = nvidia-bright-cluster-manager, nativeType = NVIDIA BCM Head Node, version = 1)

Configure and install the integration

  1. From All Clients, select a client
  2. Go to Setup > Integrations and Apps.
  3. Click Manage Apps.
    Notes:
    • If there are already installed applications, it will redirect to the INSTALLED APPS page, where all the installed applications are displayed.
    • If there are no installed applications, it will navigate to the ADD APP page.
NVIDIA Bright Cluster Manager
  1. Click + ADD on the INSTALLED APP page. The ADD APP page displays all the available applications along with the newly created application with the version.
    Note: You can even search for the application using the search option available. Also you can use the All Categories option to search.
NVIDIA Bright Cluster Manager
  1. Click ADD in the Nvidia Bright Cluster Manager application.
  2. In the Configurations page, click + ADD. The Add Configuration page appears.
  3. Enter the below mentioned BASIC INFORMATION:
FunctionalityDescription
NameEnter the name for the configuration.
BCM Manager IP AddressProvide the BCM Manager IP Address.
BCM Manager CredentialsSelect the credentials from the drop-down list.

Note: Click + Add to create a credential.
CMSH Source Path in BCM ManagerEnter the source path for the BCM Manager.

Note: By default /cm/local/apps/cmd/bin is provided.

Note: Select App Failure Notifications; if turned on, you will be notified in case of an application failure that is, Connectivity Exception, Authentication Exception.

  1. In the RESOURCE TYPE section, select:
    • ALL: All the existing and future resources will be discovered.
    • SELECT: You can select one or multiple resources to be discovered.
  2. In the DISCOVERY SCHEDULE section, select Recurrence Pattern to add one of the following patterns:
    • Minutes
    • Hourly
    • Daily
    • Weekly
    • Monthly
  3. Click ADD.
NVIDIA Bright Cluster Manager
  1. Now the configuration is saved and displayed on the configurations page after you save it.
    Note: From the same page, you may Edit and Remove the created configuration.
  2. Click NEXT
  3. In the Installation page, select an existing registered gateway profile, and click FINISH.
NVIDIA Bright Cluster Manager

The application is now installed and displayed on the Installed Integration page. Use the search field to find the installed application.

Modify the Configuration

View the NVIDIA Bright Cluster Manager details

The NVIDIA Bright Cluster Manager integration is displayed in the Infrastructure > Resources > Cluster. You can navigate to the Attributes tab to view the discovery details, and the Metrics tab to view the metric details for NVIDIA Bright Cluster Manager.

NVIDIA Bright Cluster Manager
NVIDIA Bright Cluster Manager

Supported metrics

Native TypeMetric NameDisplay NameUnitsDescriptionMetric Available from Application Version
Clusternvidia_bcm_cluster_nodeAvailabilityStatusnvidia bcm cluster nodeAvailabilityStatusNAIt monitors the availability status of each node. Below are the possible status values: UP-0, Down-1, Closed-2, installing-3, installer_failed-4, installer_rebooting-5, installer_callinginit-6, installer_unreachable-7, installer_burning-8, burning-9, unknown-10, opening-11, going_down-12, pending-13, no data-141.0.0
nvidia_bcm_cluster_slurmDaemon_statusnvidia bcm cluster slurmDaemon statusNAIt monitors the running status of slurmDaemon & allocation status of the each node, Below are the possible states: allocated-0, idle-1, down-21.0.0
nvidia_bcm_cluster_gpuUnits_closednvidia bcm cluster gpuUnits closedcountIt monitors the count of how many GPU units in closed state1.0.0
nvidia_bcm_cluster_gpuUnits_downnvidia bcm cluster gpuUnits downcountIt monitors the count of how many GPU units in Down state1.0.0
nvidia_bcm_cluster_gpuUnits_totalnvidia bcm cluster gpuUnits totalcountIt monitors the count of total GPU units1.0.0
nvidia_bcm_cluster_gpuUnits_upnvidia bcm cluster gpuUnits upcountIt monitors the count of how many GPU units are UP.1.0.0
nvidia_bcm_cluster_cpuIdlenvidia bcm cluster cpuIdle%It monitors the % of CPU Idle Time.1.0.0
nvidia_bcm_cluster_nodesTotalnvidia bcm cluster nodesTotalcountIt monitors the count of total number of nodes.1.0.0
nvidia_bcm_cluster_nodesUpnvidia bcm cluster nodesUpcountIt monitors the count of how many nodes are in UP state.1.0.0
nvidia_bcm_cluster_nodesDownnvidia bcm cluster nodesDowncountIt monitors the count of how many nodes are in DOWN state.1.0.0
nvidia_bcm_cluster_nodesClosednvidia bcm cluster nodesClosedcountIt monitors the count of how many nodes are in closed state.1.0.0
nvidia_bcm_cluster_devicesTotalnvidia bcm cluster devicesTotalcountIt monitors the count of total devices.1.0.0
nvidia_bcm_cluster_devicesUpnvidia bcm cluster devicesUpcountIt monitors the count of how many devices are in UP state.1.0.0
nvidia_bcm_cluster_devicesDownnvidia bcm cluster devicesDowncountIt monitors the count of how many devices are in DOWN state.1.0.0
nvidia_bcm_cluster_coresUpnvidia bcm cluster coresUpcountIt monitors the count of Cores which are in UP state.1.0.0
nvidia_bcm_cluster_coresTotalnvidia bcm cluster coresTotalcountIt monitors the total count of cores1.0.0
nvidia_bcm_cluster_totalUsersLoginnvidia bcm cluster totalUsersLogincountIt monitors the count of total users login.1.0.0
nvidia_bcm_cluster_totalKnownUsersnvidia bcm cluster totalKnownUserscountIt monitors the count of total known users.1.0.0
nvidia_bcm_cluster_occupationRatenvidia bcm cluster occupationRate%It monitors the Occupation rate in %.1.0.0
nvidia_bcm_cluster_cpuUtilizationnvidia bcm cluster cpuUtilization%It monitors the % of CPU Utilization1.0.0
nvidia_bcm_cluster_failedJobsnvidia bcm cluster failedJobscountReturns number of failed jobs on the cluster.1.0.0
nvidia_bcm_cluster_queuedJobsnvidia bcm cluster queuedJobscountReturns number jobs in queue on the cluster.1.0.0
nvidia_bcm_cluster_runningJobsnvidia bcm cluster runningJobscountReturns number of jobs running on the cluster.1.0.0
nvidia_bcm_cluster_nodesInQueuenvidia bcm cluster nodes InQueuecountReturns number of nodes queued on the cluster1.0.0
nvidia_bcm_cluster_uniqueUserLogincountnvidia bcm cluster uniqueUserLogincountcountReturns number of unique users logged in to the cluster.1.0.0
nvidia_bcm_cluster_smartHdaTempnvidia bcm cluster smartHdaTempcelsiusIt monitors the temperature of spindle disks2.0.0
Head Nodenvidia_bcm_headNode_healthStatusnvidia bcm headNode healthStatusNAHealth status(PASS or FAIL) of each component.1.0.0
nvidia_bcm_headNode_blockedProcessesnvidia bcm headNode blockedProcessescountBlocked processes waiting for I/O.1.0.0
nvidia_bcm_headNode_systemCpuTimenvidia bcm headNode systemCpuTimejifflesCPU time spent in system mode.1.0.0
nvidia_bcm_headNode_cpuWaitTimenvidia bcm headNode cpuWaitTimejifflesCPU time spent in I/O wait mode.1.0.0
nvidia_bcm_headNode_errorsRecvnvidia bcm headNode errorsRecvErrors per SecPackets/s received with error.1.0.0
nvidia_bcm_headNode_errorsSentnvidia bcm headNode errorsSentErrors per SecPackets/s sent with error.1.0.0
nvidia_bcm_headNode_hardwareCorruptedMemorynvidia bcm headNode hardwareCorruptedMemoryBytesHardware corrupted memory detected by ECC.1.0.0
nvidia_bcm_headNode_memoryFreenvidia bcm headNode memoryFreeGBFree system memory.1.0.0
nvidia_bcm_headNode_gpu_utilizationnvidia bcm headNode gpu utilization%Average GPU utilization percentage.1.0.0
nvidia_bcm_headNode_gpu_temperaturenvidia bcm headNode gpu temperatureCelsiusGPU temperature.1.0.0
Physical Nodenvidia_bcm_physicalNode_healthStatusnvidia bcm physicalNode healthStatusNAHealth status(PASS or FAIL) of each component.1.0.0
nvidia_bcm_physicalNode_blockedProcessesnvidia bcm physicalNode blockedProcessescountBlocked processes waiting for I/O.1.0.0
nvidia_bcm_physicalNode_systemCpuTimenvidia bcm physicalNode systemCpuTimejifflesCPU time spent in system mode.1.0.0
nvidia_bcm_physicalNode_cpuWaitTimenvidia bcm physicalNode cpuWaitTimejifflesCPU time spent in I/O wait mode.1.0.0
nvidia_bcm_physicalNode_errorsRecvnvidia bcm physicalNode errorsRecvErrors per SecPackets/s received with error.1.0.0
nvidia_bcm_physicalNode_errorsSentnvidia bcm physicalNode errorsSentErrors per SecPackets/s sent with error.1.0.0
nvidia_bcm_physicalNode_hardwareCorruptedMemorynvidia bcm physicalNode hardwareCorruptedMemoryBytesHardware corrupted memory detected by ECC.1.0.0
nvidia_bcm_physicalNode_memoryFreenvidia bcm physicalNode memoryFreeGBFree system memory.1.0.0
nvidia_bcm_physicalNode_gpu_utilizationnvidia bcm physicalNode gpu utilization%GPU utilization percentage1.0.0
nvidia_bcm_physicalNode_gpu_temperaturenvidia bcm physicalNode gpu temperatureCelsiusGPU temperature.1.0.0
nvidia_bcm_physicalNode_nfsmount_totalSizenvidia bcm physicalNode nfsmount totalSizeGBIt monitors the total size of the nfs mount file on the node.3.0.0
nvidia_bcm_physicalNode_nfsmount_usedSizenvidia bcm physicalNode nfsmount usedSizeGBIt monitors the used size of the nfs mount file on the node.3.0.0
nvidia_bcm_physicalNode_nfsmount_utilizationnvidia bcm physicalNode nfsmount utilization%It monitors the percentage utilization of the nfs mount file on the node.3.0.0

Risks, Limitations & Assumptions

  • Application can handle Critical/Recovery failure notifications for below two cases when user enables App Failure Notifications in configuration:
    • Connectivity Exception (ConnectTimeoutException, HttpHostConnectException, UnknownHostException).
    • Authentication Exception (UnauthorizedException).
  • Application will not send any duplicate/repeat failure alert notification until the already existed critical alert is recovered.
  • Application cannot control monitoring pause/resume actions based on above alerts.
  • Metrics can be used to monitor NVIDIA Bright Cluster Manager resources and can generate alerts based on the threshold values.
  • Macro replacement limitation (i.e, customization for threshold breach alert subject, description).
  • Component level thresholds can be configured on each resource level.
  • No support of showing activity log and applied time.
  • No support for the option to get the latest snapshot metric.
  • Application is not compatible with Cluster and Classic Gateway.
  • Application is not returning any data if any SSH connectivity issues (based on monitoring and discovery frequency our App will try to establish SSH connections to the target device).
  • Application will work only with SSH credentials, with ssh port 22 in the open state.
  • Application will not work for low monitoring frequencies. We recommend providing a minimum monitoring frequency of 15 minutes.
  • Virtual Nodes discovery and Monitoring will not be supported.