A DataNode stores data in HadoopFileSystem. A functional filesystem has more than one DataNode, with data replicated across them.

Prerequisite

For Virtual Machines, install the Linux Agent.

Configuring the credentials

Configure the credentials in the directory /opt/opsramp/agent/conf/app.d/creds.yaml

hdfs-datanode:
- name: hdfs-datanode
  user: <username>
  pwd: <Password>
  encoding-type: plain
  labels:
    key1: val1
    key2: val2

Configuring the application

Virtual machine

Configure the application in the directory /opt/opsramp/agent/conf/app/discovery/auto-detection.yaml

- name: hdfs-datanode
  instance-checks:
    process-check:
      - datanode
    port-check:
      - 50075
      - 9864
    uri-check:
      - "jmx?qry=Hadoop:service=DataNode,name=FSDatasetState*"
      - "jmx?qry=java.lang:type=OperatingSystem"

Docker environment

Configure the application in the directory /opt/opsramp/agent/conf/app/discovery/auto-container-detection.yaml

- name: hdfs-datanode
  container-checks:
    image-check:
      - datanode
    port-check:
      - 50075
      - 9864

Kubernetes environment

Configure the application in config.yaml

- name: hdfs-datanode
  container-checks:
    image-check:
      - datanode
    port-check:
      - 50075
      - 9864

Validate

Go to Resources under the Infrastructure tab to check if your resources are onboarded and the metrics are collected.

Metrics

OpsRamp MetricMetric Display NameUnit
hdfs_datanode_dfs_remaining

The remaining disk space left
Dfs Remainingbytes
hdfs_datanode_dfs_remaining_percent

The remaining disk space left
Dfs Remaining Percentpercent
hdfs_datanode_dfs_capacity

Capacity of the disk
Dfs Capacitybytes
hdfs_datanode_dfs_used

Disk usage
Dfs Usedbytes
hdfs_datanode_dfs_used_percent

Disk usage
Dfs Used Percentpercent
hdfs_datanode_cache_capacity

Capacity of the cache
Cache Capacitybytes
hdfs_datanode_cache_used

Cache used
Cache Usedbytes
hdfs_datanode_num_failed_volumes

Number of failed volumes
Num Failed Volumes
hdfs_datanode_last_volume_failure_date

Date/time of the last volume failure since epoch
Last Volume Failure Datemilliseconds
hdfs_datanode_estimated_capacity_lost_total

Estimated capacity lost
Estimated Capacity Lost Totalbytes
hdfs_datanode_num_blocks_cached

Number of blocks cached
Num Blocks CachedBlocks
hdfs_datanode_num_blocks_failed_to_cache

Number of blocks that failed to cache
Num Blocks Failed To CacheBlocks
hdfs_datanode_num_blocks_failed_to_uncache

Number of failed blocks to remove from cache
Num Blocks Failed To UncacheBlocks
hdfs_datanode_process_cpu_load

CPU Load of the process
Process Cpu LoadLoad