NVIDIA Bright Cluster Manager

Application Version and Upgrade Details

Introduction

NVIDIA Bright Cluster Manager offers fast deployment, and end-to-end management for heterogeneous high-performance computing (HPC) and AI server clusters at the edge in the data center, and in multi/hybrid-cloud environments. It automates provisioning and administration for clusters ranging in size from a couple of nodes to hundreds of thousands. It also supports CPU-based and NVIDIA GPU-accelerated systems, and enables orchestration with Kubernetes.

Monitoring Use cases

• In case of any threshold breach or unexpected metric behavior, based on configurations the device monitoring helps to collect the metric values with respect to time and sends alerts to the intended customer team to act up immediately.
• It helps the customer with smooth functioning of business with minimal or zero downtime in case of any infrastructure related issues occurring.

Resource Hierarchy

NVIDIA BrightCluster Manager
            → NVIDIA BCM Head Node
                → NVIDIA BCM Virtual Node
                → NVIDIA BCM Physical Node
                → NVIDIA BCM Linux Server

Prerequisites

• OpsRamp Nextgen Gateway supported version 14.0.0 and above.
• OpsRamp Classic Gateway supported version 14.0.0 and above.
  Note: OpsRamp recommends using the latest Gateway version for full coverage of recent bug fixes, enhancements, etc.

Default Monitoring Configurations

Bright Cluster Manager has default Global Device Management Policies, Global Templates, Global Monitors and Global Metrics in OpsRamp. You can customize these default monitoring configurations as per your business use cases by cloning respective Global Templates, and Global Device Management Policies. OpsRamp recommends doing this activity before installing the application to avoid noise alerts and data.

1. Default Global Device Management Policies

   OpsRamp has a Global Device Management Policy for each Native Type of NVIDIA Bright Cluster Manager. You can find those Device Management Policies at Setup > Resources > Device Management Policies, search with suggested names in global scope. Each Device Management Policy follows below naming convention:

   {appName nativeType - version}

   Ex: NVIDIA BrightCluster Manager NVIDIA BCM Head Node (i.e, appName = nvidia-bright-cluster-manager, nativeType = NVIDIA BCM Head Node, version = 1)

2. Default Global Templates

   OpsRamp has a Global Template for each Native Type of NVIDIA Bright Cluster Manager. You can find those templates at Setup > Monitoring > Templates, search with suggested names in global scope. Each template follows below naming convention:

   {appName nativeType 'Template' - version}

   Ex : nvidia-bright-cluster-manager NVIDIA BCM Head Node Template - 1 (i.e, appName = nvidia-bright-cluster-manager, nativeType = NVIDIA BCM Head Node Template, version = 1)

3. Default Global Monitors

   OpsRamp has a Global Monitors for each Native Type which has monitoring support. You can find those monitors at Setup > Monitoring > Monitors, search with suggested names in global scope. Each Monitors follows below naming convention:

   {monitorKey appName nativeType - version}

   Ex: NVIDIA BCM Head Node Monitor nvidia-bright-cluster-manager NVIDIA BCM Head Node Container 1(i.e, monitorKey = NVIDIA BCM Head Node Monitor, appName = nvidia-bright-cluster-manager, nativeType = NVIDIA BCM Head Node, version = 1)

Supported Metrics

Configure and Install the NVIDIA Bright Cluster Manager Integration

1. From All Clients, select a client.
2. Go to Setup > Account.
3. Select the Integrations and Apps tab.
4. The Installed Integrations page, where all the installed applications are displayed.
   Note: If there are no installed applications, it will navigate to the Available Integrations and Apps page.
5. Click + ADD on the Installed Integrations page. The Available Integrations and Apps page displays all the available applications along with the newly created application with the version.
   Note: Search for the application using the search option available. Also you can use the All Categories option to search.

6. Click ADD in the Nvidia Bright Cluster Manager tile.
7. In the Configurations page, click + ADD. The Add Configuration page appears.
8. Enter the below mentioned BASIC INFORMATION:

Notes:

• BCM Manager IP Address should be accessible from Gateway.
• Select App Failure Notifications; if turned on, you will be notified in case of an application failure that is, Connectivity Exception, Authentication Exception.

9. Select the below mentioned Custom Attribute:

Note: The custom attribute that you add here will be assigned to all the resources that are created by the integration. You can add a maximum of five custom attributes (key and value pair).

10. In the RESOURCE TYPE section, select:

    • ALL: All the existing and future resources will be discovered.
    • SELECT: You can select one or multiple resources to be discovered.
      
      

11. In the DISCOVERY SCHEDULE section, select Recurrence Pattern to add one of the following patterns:

    • Minutes
    • Hourly
    • Daily
    • Weekly
    • Monthly
      
      

12. Click ADD.

Now the configuration is saved and displayed on the configurations page after you save it.
Note: From the same page, you may Edit and Remove the created configuration.

13. Click NEXT.

14. Below are the optional steps you can perform on the Installing page.

• Under the ADVANCED SETTINGS, Select the Bypass Resource Reconciliation option, if you wish to bypass resource reconciliation when encountering the same resources discovered by multiple applications.

  Note: If two different applications provide identical discovery attributes, two separate resources will be generated with those respective attributes from the individual discoveries.

• Click +ADD to create a new collector by providing a name or use the pre-populated name.

15. Select an existing registered profile.

16. Click FINISH.

The integration is now installed and displayed on the Installed Integration page. Use the search field to find the installed integration.

Modify the Configuration

• See Modify an Installed Integration or Application article.

  Note: Select the Nvidia Bright Cluster Manager application.

View the NVIDIA Bright Cluster Manager Details

The NVIDIA Bright Cluster Manager integration is displayed in the Infrastructure > Resources > Cluster. You can navigate to the Attributes tab to view the discovery details, and the Metrics tab to view the metric details for NVIDIA Bright Cluster Manager.

Resource Filter Input keys

Supported Alert Custom Macros

Customize the alert subject and description with below macros then it will generate alert based on customisation.
Supported macros keys:

Risks, Limitations & Assumptions

• Application can handle Critical/Recovery failure notifications for below two cases when user enables App Failure Notifications in configuration:
  • Connectivity Exception (ConnectTimeoutException, HttpHostConnectException, UnknownHostException).
  • Authentication Exception (UnauthorizedException).
• Application will not send any duplicate/repeat failure alert notification until the already existed critical alert is recovered.
• Application cannot control monitoring pause/resume actions based on above alerts.
• Metrics can be used to monitor NVIDIA Bright Cluster Manager resources and can generate alerts based on the threshold values.
• No support of showing activity log and applied time.
• This application supports both Classic Gateway and NextGen Gateway.
• Application is not returning any data if any SSH connectivity issues (based on monitoring and discovery frequency our App will try to establish SSH connections to the target device).
• Application will work only with SSH credentials, with ssh port 22 in the open state.
• Virtual Nodes discovery and Monitoring will not be supported.
• To discover and monitor the NVIDIA BCM Linux Server and its metrics, sshpass should be installed on the BCM Cluster. If the sshpass package is not available, ensure that the credentials of both the BCM cluster and the Linux Server are the same.