Pod Crashed due to high CPU Utilization

Folow the below steps to debug the high CPU utilization:

1. Get the process id of vprobe container using the following command.

   kubectl exec -it nextgen-gw-0 -c vprobe -- /bin/bash -c 'ps -ef | grep vprobe'

   Example:

   ubuntu@ashok:~$ kubectl exec -it nextgen-gw-0 -c vprobe -- /bin/bash -c 'ps -ef | grep vprobe' 
   gateway        1       0  0 15:02 ?        00:00:00 /bin/bash /opt/gateway/vprobe/bin/vprobe.sh 
   gateway        6       1  6 15:02 ?        00:03:20 java -Djava.naming.provider.url=file:/opt/gateway/vprobe/temp/jndi -Dserver.home= -Dsun.net.inetaddr.ttl=0 -Djava.net.preferIPv4Stack=true -Dhttp.nonProxyHosts=localhost -Dhttps.protocols=TLSv1,TLSv1.1,TLSv1.2,SSLv3 -Djdk.tls.client.protocols=TLSv1,TLSv1.1,TLSv1.2,SSLv3 --add-opens java.base/java.util.concurrent=ALL-UNNAMED --add-opens java.base/sun.security.ssl=ALL-UNNAMED -XX:InitialRAMPercentage=30 -XX:MaxRAMPercentage=80 -XX:+ExitOnOutOfMemoryError -XX:+UnlockDiagnosticVMOptions -XX:+LogVMOutput -XX:LogFile=/var/log/app/vprobe-jvm-thread.dump -classpath /opt/gateway/vprobe/conf:/opt/gateway/vprobe/lib/:/opt/gateway/vprobe/lib/* com.vistara.gateway.core.VistaraGateway 
   gateway     1288       0  0 15:52 pts/0    00:00:00 /bin/bash -c ps -ef | grep vprobe 

   Note

   Here, 6 is the vprobe process id.
2. Get the list of child threads of vprobe process id using the following command.

   kubectl exec -it nextgen-gw-0 -c vprobe -- /bin/bash -c 'top -n 2 -b -H -p <process_id>' > /tmp/child_threads.txt

   Example:

   kubectl exec -it nextgen-gw-0 -c vprobe -- /bin/bash -c 'top -n 2 -b -H -p 6' > /tmp/child_threads.txt

   Note

   Here, 2 is the number of iterations and 6 is the vprobe process id. You can find the output in /tmp/child_threads.txt
3. Generate the thread dump for vprobe using jstack, use the following command.

   kubectl exec -it nextgen-gw-0 -c vprobe -- /bin/bash -c '/jcmd/bin/jstack <process_id> > /tmp/thread_dump.dump'

   Example:

   kubectl exec -it nextgen-gw-0 -c vprobe -- /bin/bash -c '/jcmd/bin/jstack 6 > /tmp/thread_dump.dump'

   Note

   Here, 6 is the vprobe process id.
4. Transfer the thread dump from the vprobe container to the gateway node, use the following command.

   kubectl cp nextgen-gw-0:<container path> -c vprobe -n default <node path> 

   Example:

   kubectl cp nextgen-gw-0:/tmp/thread_dump.dump -c vprobe -n default /tmp/thread_dump.dump

   Note

   Here, 6 is the vprobe process id.
5. Share the child threads and thread dump files to the OpsRamp Support Team.