Processes in Action

If you are like me, knowing how things work in theory is not enough. You want to see how things work on your system. Linux provides several tools for you to watch what is happening. The first tool is perhaps the only one that the majority of users have ever seen. This is the ps command, which gives you the process status of particular processes.

Although users can look at processes using the ps command, they cannot look at the insides of the processes themselves. This is because the ps command simply reads the process table, which contains only the control and data structures necessary to administer and manager the process and not the process itself. Despite this, using ps can not only show you a lot about what your system is doing but can give you insights into how the system works. Because much of what I will talk about is documented in the ps man-page, I suggest in advance that you look there for more details.

If you start ps from the command with no options, the default behavior is to show the processes running for the user who ran the ps. With a logon on two terminals, it looks something like this:

PID TTY TIME CMD 1857 pts/5 00:00:00 su 1858 pts/5 00:00:00 bash 24375 pts/6 00:00:00 su 24376 pts/6 00:00:00 bash 25620 pts/6 00:00:00 ps

This shows the process ID (PID), the terminal that the process is running on (TTY), the total amount of time the process has had on the CPU (TIME), and the command that was run (CMD).

The PID is assigned to the process when it is created. This is a unique number that starts at 1 with the init process when the system is first booted, and then increments until it reaches a pre-defined limit and then it starts over. Note that if a particular process is still running, the system will not re-use that ID, but will skip to the next free ID.

In the example above, the processes are actually running on a pseudo-terminal. In a nutshell, this means that the terminal is not really a physical device. You typically have pseudo-terminals when opening a console within a GUI or conneting to a remote machine via telent or ssh.

Note that I have read some books that claim that since ps is showing only the processes for the current user, the terminal should always be the same. Although this appears logical at first, it is not entirely correct. If I were to be running a GUI, I could have many console sessions open, each running on their own pseudo-terminal. So, as we see in this example, there are different processes running on different a pseudo-terminal.

However, newer versions of ps that I have worked with only show the processes running on the current pseudo-terminal. If you want all of the process for a given user, you can run ps like this:

ps -u jimmo

The time is listed in hours, minutes and seconds. It is possible that the process has been on the system for hours and even days and it still reports zero time. The reason is that this is the amount of time that the process is actually running. If, for example, I were to immediately start vi when I logged in, I could continue to use vi for hours and my original shell process would not get a chance to run again. It would be sleeping until the vi process finished.

Although this is useful in many circumstances, it doesn’t say much about these processes, Lets see what the long output looks like. This is run like this:

ps -l

F S UID PID PPID C PRI NI ADDR SZ WCHAN TTY TIME CMD 100 S 501 1857 1844 0 69 0 – 537 wait4 pts/5 00:00:00 su 000 S 501 1858 1857 0 69 0 – 683 read_c pts/5 00:00:00 bash 100 S 501 24375 24362 0 69 0 – 537 wait4 pts/6 00:00:00 su 000 S 501 24376 24375 0 72 0 – 683 wait4 pts/6 00:00:00 bash 000 R 501 25621 24376 0 76 0 – 729 – pts/6 00:00:00 ps

This output looks a little better.
At least there are more entries, so maybe it is more interesting. The columns
PID, TTY, TIME, and COMMAND are the same as in the previous example. The first column (F)
is the octal representation of what flags the process has. Most have no flags, but in this
example, both su processes have a flag value of 100, which indciates super-user privileges.
This makes sense because I used su to switch to root and then used su again to switch to another
user. For a list of possible flags, see the ps man-page.

Note that example this assumes the behaviour of ps that displays all of the processes for the user across all pseudo-terminals. You may need to use the -u USERNAME option to see all of the processes.

The S column tells us what state the process is in. All but the last ps process are in state S, or sleeping. This is the typical state when they are waiting on something. However, ps must be running to give us this output, so it has a state R, for running. Note that on a single processor system, only one process can be running at once. Therefore, you will never see more than one process in state R.

Here we see that the bash process on line 5 is sleeping. Although I can’t tell from this output, I know that the event, on which the shell is waiting, is the completion of the ps command. The PID and PPID columns, the Process ID and Parent Process ID, respectively, are one indication. Notice that the PPID of the ps process is the same as the PID of the bash process. This is because I started the ps command from the bash command line and the bash had to do a fork() -exec() to start up the ps. This makes ps a child process of the bash. Because I didn’t start the ps in the background, I know the bash is waiting on the completion of the ps. (More on this in a moment.)

The C column is the processor utilization. With the exception of the ps command, all of the other processes listed are waiting for some event, such as another process to exit or (as in the case of shell processes) are waiting for user input. Note that in this example all of the entries have a 0 (zero) in the C column. This does not mean they have used no CPU time, but rather it is so low that it is reported as zero.

The PRI column shows us the priority of the process followed by the nice value (NI). The nice
value is a way of influencing how often the process is scheduled to run. The higher the nice
value the “nicer” the processes is, and lets other processes run. The SZ column is the size in
blocks of the core image of the process. This is followed by the process’ wait channel, or what the process
is waiting on. In the case of the bash process on the second line, it is waiting on the wait channel
“read_c”. That is, it is waiting to read a character from the keyboard.

The ps process is on the processor and in the state R-RUNNABLE. As a matter of fact, I have never run a ps command where ps was not RUNNABLE. Why? Well, the only way for ps to read the process table is to be running and the only way for a process to be running is to be runnable. However, with this output of ps, we cannot actually see the state, but guess that ps is running in that it is not waiting on anything (there is a – in the WCHAN column). To see what state the process is in we need to us a different option, for example ps ux, which gives us this:

USER PID %CPU %MEM VSZ RSS TTY STAT START TIME CMD linux-tu 1857 0.0 0.0 2148 4 pts/5 S Mar27 0:00 su – linux-tu linux-tu 1858 0.0 0.0 2732 4 pts/5 S Mar27 0:00 -bash linux-tu 24375 0.0 0.2 2148 824 pts/6 S 16:36 0:00 su – linux-tu linux-tu 24376 0.0 0.3 2732 1484 pts/6 S 16:36 0:00 -bash linux-tu 25659 0.0 0.3 2452 1488 pts/6 R 21:52 0:00 ps -ux

Because I am running these processes as the user linux-tu, the USER column shows this. The owner is almost always the user who started the process. However, you can change the owner of a process using the setuid() or the seteuid() system call.

VSZ, is the size of the virtual memory image in kilobytes. This is followed by The RSS column is the “resident set size,” or how many kilobytes of the program is in memory. The difference being that the virtual memory size is essentially what the total memory size of the process would be if it were all in memory, while the RSS is what is actually in memory.

Though I can’t prove this, I can make some inferences. First, let’s look
at the ps output again. This time, lets start ps -l in the background, which gives
us the following output:

100 S   501  1857  1844  0  69   0 -   537 wait4  pts/5    00:00:00 su
000 S   501  1858  1857  0  69   0 -   683 read_c pts/5    00:00:00 bash
100 S   501 24375 24362  0  69   0 -   537 wait4  pts/6    00:00:00 su
000 S   501 24376 24375  0  72   0 -   684 read_c pts/6    00:00:00 bash
000 R   501 29043 24376  0  78   0 -   729 -      pts/6    00:00:00 ps

We now see that the same process (24376) has a different wait channel. This time it is read_c, which
tells us it is waiting for keyboard input.

In the second example, bash did a fork-exec, but because we put it in the background, it returned to the prompt and didn’t wait for the ps to complete. Instead, it was waiting for more input from the keyboard. In the second example, bash did nothing. However, in the first example, we did not put the command in the background, so the WCHAN was the completion of ps.

So far we have only seen the “short” version of the command. In essence, this is only the actual command name. If we use the -f (for “full”) we can see the command and any options, like this:

UID PID PPID C STIME TTY TIME CMD jimmo 2271 2184 0 Mar17 pts/11 00:00:00 /bin/bash jimmo 17344 2271 0 10:03 pts/11 00:00:00 ps -f

Note that in the case of the bash command, we see the full path. This is because that is what was used to start the process. Without the -f all we would see is bash. In the case of the ps command, the full listing shows the option -f as well. However, without it, all we would see is ps.

To see the relationship between various processes you can use the –forest. If we wanted to see a forest (i.e. many process “trees”) of the user jimmo, the command might look like this:

ps -u jimmo –forest

Which gives us output like this:

PID TTY TIME CMD 2184 ? 00:00:37 _ kdeinit 2260 pts/10 00:00:00 | _ bash 17058 pts/10 00:00:00 | | _ man 17059 pts/10 00:00:00 | | _ sh 17063 pts/10 00:00:00 | | _ nroff 17069 pts/10 00:00:00 | | | _ groff 17071 pts/10 00:00:00 | | | | _ grotty 17072 pts/10 00:00:00 | | | _ cat 17064 pts/10 00:00:00 | | _ less 11607 ? 00:00:18 _ soffice.bin 11643 ? 00:00:00 | _ soffice.bin 11644 ? 00:00:00 | _ soffice.bin 11645 ? 00:00:00 | _ soffice.bin

Note that this is only an portion of the actual process forest. At this moment on my machine I literally have dozens of processes. However, this example gives you a good idea of what a process looks like. Give it a try on your system to see what you might be running.

One interesting thing to note is the tree starting at PID 17058. This is all part of a man command that I started. For the most part, I am not even aware that these other process (i.e. nroff are running).

Using the -a option, you show all of the processes for the given process
with the exception of the so-call “session leader”. In a nutshell, the
session leader is the process which started the session. Typically, this is the login
shell or often kdeinit if you are running a console from within KDE
(for example).

If you did a listing of all processes using, for example, ps -e you might get something like this:

PID TTY TIME CMD 1 ? 00:00:05 init 2 ? 00:00:00 keventd 3 ? 00:00:00 ksoftirqd_CPU0 4 ? 00:00:09 kswapd 5 ? 00:00:00 bdflush 6 ? 00:00:00 kupdated 7 ? 00:00:05 kinoded 8 ? 00:00:00 mdrecoveryd 13 ? 00:00:00 kjournald 387 ? 00:00:00 lvm-mpd 435 ? 00:00:00 kjournald 440 ? 00:00:00 pagebufd 441 ? 00:00:00 xfslogd/0 442 ? 00:00:00 xfsdatad/0 443 ? 00:00:00 xfssyncd 444 ? 00:00:00 xfssyncd 445 ? 00:00:07 kjournald 446 ? 00:00:00 xfssyncd 447 ? 00:00:00 kjournald 448 ? 00:00:00 kjournald 923 ? 00:00:01 syslogd 926 ? 00:00:02 klogd 985 ? 00:00:00 khubd 1130 ? 00:00:00 resmgrd 1166 ? 00:00:00 portmap 1260 ? 00:00:00 mysqld_safe 1325 ? 00:00:00 acpid 1340 ? 00:00:00 mysqld 1352 ? 00:00:00 sshd 1474 ? 00:00:00 cupsd 1479 ? 00:00:00 mysqld 1480 ? 00:00:00 mysqld 1481 ? 00:00:00 mysqld 1482 ? 00:00:00 mysqld 1676 ? 00:00:00 master 1747 ? 00:00:00 qmgr 1775 ? 00:00:00 httpd 1842 ? 00:00:00 cron 1846 ? 00:00:00 nscd 1847 ? 00:00:00 nscd 1848 ? 00:00:00 nscd 1849 ? 00:00:00 nscd 1850 ? 00:00:00 nscd 1851 ? 00:00:00 nscd 1852 ? 00:00:00 nscd 1856 ? 00:00:00 smbd 1876 ? 00:00:44 httpd 1972 ? 00:00:00 kdm 2022 ? 01:10:11 X 2023 ? 00:00:00 kdm 2026 tty1 00:00:00 mingetty 2027 tty2 00:00:00 mingetty 2028 tty3 00:00:00 mingetty 2029 tty4 00:00:00 mingetty 2030 tty5 00:00:00 mingetty 2031 tty6 00:00:00 mingetty 2058 ? 00:00:00 kde 2094 ? 00:00:00 gpg-agent 2095 ? 00:00:00 ssh-agent 2158 ? 00:00:01 kamix 2160 ? 00:00:14 kdeinit 2164 ? 00:00:00 kdeinit 2166 ? 00:00:01 susewatcher 2167 ? 00:03:54 suseplugger 2178 ? 00:00:07 kdeinit 2180 ? 00:04:33 kmail 2182 ? 00:00:05 kscd 2184 ? 00:00:38 kdeinit 2187 ? 00:08:20 quanta 2188 pts/3 00:00:00 bash 2191 pts/5 00:00:00 bash 2199 pts/8 00:00:00 bash 2220 pts/9 00:00:00 bash 2224 ? 00:01:10 quanta 2260 pts/10 00:00:00 bash 2271 pts/11 00:00:00 bash 2273 pts/13 00:00:00 bash 2277 pts/14 00:00:00 bash 2314 ? 00:00:00 kalarmd 2402 ? 00:00:35 httpd 2831 ? 00:00:01 kdeinit 10244 ? 00:00:00 kdeinit 11607 ? 00:00:18 soffice.bin 11643 ? 00:00:00 soffice.bin 11644 ? 00:00:00 soffice.bin 11692 ? 00:02:21 kdeinit 12035 ? 00:00:10 kdeinit 12036 pts/4 00:00:00 bash 12058 pts/6 00:00:00 bash 12088 ? 00:00:58 kdeinit 12148 pts/7 00:00:00 bash 12238 pts/7 00:00:00 man 12239 pts/7 00:00:00 sh 12240 pts/7 00:00:00 less 12241 pts/7 00:00:00 gzip 12242 pts/7 00:00:00 zsoelim 12243 pts/7 00:00:00 tbl 12244 pts/7 00:00:00 nroff 12249 pts/7 00:00:00 groff 12250 pts/7 00:00:00 cat 12251 pts/7 00:00:00 troff 12252 pts/7 00:00:00 grotty 12260 pts/12 00:00:00 bash 12300 pts/15 00:00:00 bash 13010 pts/6 00:00:00 man 13011 pts/6 00:00:00 sh 13016 pts/6 00:00:00 less 13979 ? 00:00:00 smbd 16049 ? 00:00:08 smbd 16138 pts/3 00:00:00 man 16139 pts/3 00:00:00 sh 16144 pts/3 00:00:00 less 16251 ? 00:00:00 mysqld 16273 ? 00:00:00 mysqld 16387 ? 00:00:48 grip 16388 ? 00:00:00 grip 16506 ? 00:00:11 kdeinit 16574 ? 00:00:00 pickup 16683 ? 00:00:00 smbd 16883 ? 00:00:00 login 16922 ? 00:00:00 kdeinit 17058 pts/10 00:00:00 man 17059 pts/10 00:00:00 sh 17063 pts/10 00:00:00 nroff 17064 pts/10 00:00:00 less 17069 pts/10 00:00:00 groff 17071 pts/10 00:00:00 grotty 17072 pts/10 00:00:00 cat 17165 pts/11 00:00:00 ps

Note that in many cases, there is no terminal associated with a specific process. This is common for system processes (i.e. daemons) as there is no real terminal associated with it.

As a side note, consider that a terminal may belong to a process as its “controlling terminal”. This a unique terminal device associated with the process. Or better yet, associated with the session. Each process of that sessions will have the same controlling terminal. Although a session or process can have only a single controlling terminal, a process does not need to have a controlling terminal at all. Without going into details of UNIX programming, we can simply say that a process can break that connection and run with a controlling terminal as is common for graphical processes. These processes can start child processes, which also do not have a controlling terminal. YOu can see in the output here, that process without a crontroll terminal have a question-mark (?) in the TTY column.

Another thing to note is that system processes that are started when the system was first loaded have very low PIDs. On some system, there may be so little going on that all such system processes have single digit PIDs (i.e. less than 100). However, in the example above, there are a number of system processes with relatively high PIDs (i.e. close to 1000). All this means is that so many processes were started (and probably ended immediately).

For the most part, all of the processes on this system that are associated with a terminal are associated with a pseudo-terminal. However, you will see a number of mingetty processes running on physical terminals tty1 through tty6. These represent the virtual consoles and although no user is using them at the moment, there is a process associated with them.

Note that by default mingetty runs on terminals tty1 through tty6, but these still need to be explicitely actived in the /etc/inittab file. For details on the /etc/inittab look at the section on run-levels.

Similar to the –forest option to ps is the pstree command. If we wanted to highlight the current process and all of it’s ancestors we would use the -h option, which would give us something like this:

init-+-acpid |-bdflush |-cron |-cupsd |-httpd---2[httpd] |-kalarmd |-kamix |-kdeinit-+-artsd | |-grip---grip | |-18[kdeinit] | |-kdeinit-+-bash---man---sh---less | | |-5[bash] | | |-bash---man---sh-+-less | | | -nroff-+-cat | | |-groff---grotty | | <strong>-bash---pstree</strong> | |-kdeinit-+-3*[bash] | | |-bash---man---sh---less | |-bash---man-+-gzip | | |-less | | -sh-+-nroff-+-cat | | |-groff-+-grotty | | | -troff | | |-tbl | |-zsoelim | |-2[quanta] | -soffice.bin---soffice.bin---4*[soffice.bin] |-11*[kdeinit] |-kdm-+-X |-kdm---kde-+-gpg-agent | |-kwrapper | -ssh-agent |-keventd |-khubd |-kinoded |-8*[kjournald] |-klogd |-kmail |-kscd |-ksoftirqd_CPU0 |-kswapd |-kupdated |-login---bash |-lvm-mpd |-master-+-pickup |-qmgr |-mdrecoveryd |-5[mingetty] |-mysqld_safe---mysqld---mysqld---10[mysqld] |-nscd---nscd---5[nscd] |-pagebufd |-portmap |-resmgrd |-smbd---3[smbd] |-sshd |-suseplugger |-susewatcher |-syslogd |-xfsdatad/0 |-xfslogd/0 `-3*[xfssyncd]

Note that the processes here are sorted alphabetically. You can use the -n option to sort the tree by process id.

You can find more details about intepreting this output in the
section on monitoring processing.