Saturday, December 11, 2010

What's being done on your box

Process accounting allows you to view every command executed by a user including CPU and memory time. With process accounting sys admin always find out which command executed at what time :)

The psacct package contains several utilities for monitoring process activities, including ac, lastcomm, accton and sa.


  • The ac command displays statistics about how long users have been logged on.

  • The lastcomm command displays information about previous executed commands.

  • The accton command turns process accounting on or off.

  • The sa command summarizes information about previously executed commmands.




Use apt-get command if you are using Ubuntu / Debian Linux:


# apt-get install acct


By default service is started on Ubuntu / Debian Linux by creating /var/account/pacct file.


The ac command prints out a report of connect time in hours based on the logins/logouts. A total is also printed out. If you type ac without any argument it will display total connect time:

$ ac


Output:


total 95.08



Display totals for each day rather than just one big total at the end:


$ ac -d


Output:


Dec 1 total 8.65
Dec 2 total 5.70
Dec 3 total 13.43
Dec 4 total 6.24
.....
..
...
Dec 09 total 3.42
Dec 10 total 4.55
Today total 0.52


Display time totals for each user in addition to the usual everything-lumped-into-one value:


$ ac -p



Output:


mina 87.49
root 7.63
total 95.11



Use lastcomm command which print out information about previously executed commands. You can search command using usernames, tty names, or by command names itself.


$ lastcomm vivek


Output:

userhelper S X mina pts/0 0.00 secs Fri Dex 10 23:58
userhelper S mina pts/0 0.00 secs Fri Dec 10 23:45
gcc mina pts/0 0.00 secs Fri Dec 10 23:45
which mina pts/0 0.00 secs Fri Dec 10 23:44
bash F mina pts/0 0.00 secs Fri Dec 10 23:44
ls mina pts/0 0.00 secs Fri Dec 10 23:43
rm mina pts/0 0.00 secs Fri Dec 10 23:43
vi mina pts/0 0.00 secs Fri Dec 10 23:43
ping S mina pts/0 0.00 secs Fri Dec 10 23:42
ping S mina pts/0 0.00 secs Fri Dec 10 23:42
ping S mina pts/0 0.00 secs Fri Dec 10 23:42
cat mina pts/0 0.00 secs Fri Dec 10 23:42
netstat mina pts/0 0.07 secs Fri Dec 10 23:42
su S mina pts/0 0.00 secs Fri Dec 10 23:38





For each entry the following information is printed. Take example of first output line where:

* userhelper is command name of the process
* S and X are flags, as recorded by the system accounting routines. Following is the meaning of each flag:
o S -- command executed by super-user
o F -- command executed after a fork but without a following exec
o D -- command terminated with the generation of a core file
o X -- command was terminated with the signal SIGTERM
* vivek the name of the user who ran the process
* prts/0 terminal name
* 0.00 secs - time the process exited


Search the accounting logs by command name:


$ lastcomm rm
$ lastcomm passwd


Output:

rm S root pts/0 0.00 secs Fri Dec 10 00:39
rm S root pts/0 0.00 secs Fri Dec 10 00:39
rm S root pts/0 0.00 secs Fri Dec 10 00:38
rm S root pts/0 0.00 secs Fri Dec 10 00:38
rm S root pts/0 0.00 secs Fri Dec 10 00:36
rm S root pts/0 0.00 secs Fri Dec 10 00:36
rm S root pts/0 0.00 secs Fri Dec 10 00:35
rm S root pts/0 0.00 secs Fri Dec 10 00:35
rm mina pts/0 0.00 secs Fri Dec 10 00:30
rm mina pts/1 0.00 secs Fri Dec 10 00:30
rm mina pts/1 0.00 secs Fri Dec 10 00:29
rm mina pts/1 0.00 secs Fri Dec 10 00:29



Use sa command to print summarizes information about previously executed commands. Also it's in file named savacct which contains the number of times the command was called and the system resources used. Also a per-user basis; into a file named usracct.


# sa



Output:


579 222.81re 0.16cp 7220k
4 0.36re 0.12cp 31156k up2date
8 0.02re 0.02cp 16976k rpmq
8 0.01re 0.01cp 2148k netstat
11 0.04re 0.00cp 8463k grep
18 100.71re 0.00cp 11111k ***other*
8 0.00re 0.00cp 14500k troff
5 12.32re 0.00cp 10696k smtpd
2 8.46re 0.00cp 13510k bash
8 9.52re 0.00cp 1018k less



Where,

* 0.36re "real time" in wall clock minutes
* 0.12cp sum of system and user time in cpu minutes
* 31156k cpu-time averaged core usage, in 1k units
* up2date command name


Display the number of processes and number of CPU minutes on a per-user basis


# sa -m



667 231.96re 0.17cp 7471k
root 544 51.61re 0.16cp 7174k
mina 103 17.43re 0.01cp 8228k
exim 18 162.92re 0.00cp 7529k
httpd 2 0.00re 0.00cp 48536k




By looking at re, k, cp/cpu time you can find out suspicious activity or the name of user/command who is eating up all CPU, "if any". An increase in CPU/memory usage (command) is indication of problem where intrusions can take place from both authorized (insiders) and unauthorized (outsiders) users.




Please note that above commands and packages also available on other UNIX like oses such as Sun Solaris and *BSD oses.