This page provides guidance to handle orphaned SerialUI processes and control the growth of the serialui_errors.log file on the OpsRamp Gateway. If left unmanaged, excessive log growth can consume disk space and lead to the gateway entering Read‑Only mode.

Problem

In some scenarios, SerialUI terminal sessions may be closed without using the exit command. When this occurs:

  • SerialUI processes may continue running in the background.
  • The log file can grow continuously.
  • Disk space exhaustion may occur, potentially causing the gateway filesystem to switch to Read‑Only mode.

As a result,

  • Gateway disk space may be consumed unexpectedly.
  • Monitoring and other gateway services may be affected.
  • Manual intervention may be required to restore normal operation.

Resolution

To mitigate this issue, a cron job is set up that

  • Regularly checks and terminates orphaned SerialUI Perl processes.
  • Rotates the serialui_errors.log file if it exceeds 100MB.
  • Posts syslog messages when actions are taken.

Procedure

Step 1: Log in to the Gateway

Log in to the gateway as the ruser user.

Step 2: Edit the Cron Jobs

Open the crontab editor:

sudo crontab -e

Step 3: Add the Monitoring Script

Add the following entry at the end of the crontab file, then save and exit:

11,31,51 * * * * pgrep -f "/usr/bin/perl /home/admin/serialUI/SeismicSerialUI.pl" | while read p; do t=$(ps -o tty= -p $p|tail -1); pp=(ps -o ppid= -p $p|tail -1); [ "$t" = "?" ] && [ "$pp" != "1" ] && kill -9 $p && logger "Terminated SerialUi Perl process as TTY was closed (PID=$p, PPID=$pp). Session could not be killed gracefully due to terminal window being closed without 'exit'."; done; l=/home/admin/logs/serialui_errors.log; b=$l.1; [ -f $l ] && s=$(stat -c\%s $l) && [ $s -gt 104857600 ] && tail -c 10485760 $l >$b && : >$l && logger "Log rotation completed: last 10MB of $l archived to $b; original log emptied." > /dev/null 2>&1

What the Script Does

  • Runs at 11, 31, and 51 minutes every hour.
  • Finds and kills orphaned SerialUI Perl processes (those with TTY ? and PPID not 1).
  • Rotates the serialui_errors.log file if it exceeds 100MB, keeping the last 10MB as an archive.
  • Posts syslog messages for both process termination and log rotation.

Step 4: Verify Cron Job Execution

To confirm that the cron job is running and taking action, monitor cron logs:

journalctl -u cron -f

Validation and Expected Behavior

  • Orphaned processes appear as follows:

        ps -ef | grep perl
    admin    2267291 2267290 19 05:46 ?        00:00:14 /usr/bin/perl /home/admin/serialUI/SeismicSerialUI.pl
    admin    2267408 2267407 19 05:46 ?        00:00:13 /usr/bin/perl /home/admin/serialUI/SeismicSerialUI.pl

    The ? in the TTY column indicates a closed terminal.

  • When the cron job runs, you should see syslog messages like:

        Nov 28 06:32:01 opsramp root: Terminated SerialUi Perl process as TTY was closed (PID=136178, PPID=136175). Session could not be killed gracefully due to terminal window being closed without 'exit'.
    Dec  1 05:51:01 root: Terminated SerialUi Perl process as TTY was closed (PID=2369852, PPID=2369846).

  • Log rotation messages will also appear in syslog when the log file is rotated.

  • Use journalctl -u cron -f to confirm the cron job is triggering as expected.

Notes

  • This workaround is validated and safe for repeated use.
  • The script ensures both process cleanup and log file management, reducing manual intervention and preventing disk space issues.