Due to the inherent complexity of being a demand-paged virtual
memory operating system, monitoring memory-related resources under
Red Hat Enterprise Linux can be confusing. Therefore, it is best to
start with the more straightforward tools, and work from there.
Using free, it is possible to get a
concise (if somewhat simplistic) overview of memory and swap
utilization. Here is an example:
total used free shared buffers cached
Mem: 1288720 361448 927272 0 27844 187632
-/+ buffers/cache: 145972 1142748
Swap: 522104 0 522104
|
We note that this system has 1.2GB of RAM, of which only about
350MB is actually in use. As expected for a system with this much
free RAM, none of the 500MB swap partition is in use.
Contrast that example with this one:
total used free shared buffers cached
Mem: 255088 246604 8484 0 6492 111320
-/+ buffers/cache: 128792 126296
Swap: 530136 111308 418828
|
This system has about 256MB of RAM, the majority of which is in
use, leaving only about 8MB free. Over 100MB of the 512MB swap
partition is in use. Although this system is certainly more limited
in terms of memory than the first system, to determine if this
memory limitation is causing performance problems we must dig a bit
deeper.
Although more cryptic than free,
vmstat has the benefit of displaying more
than memory utilization statistics. Here is the output from
vmstat 1 10:
procs memory swap io system cpu
r b w swpd free buff cache si so bi bo in cs us sy id
2 0 0 111304 9728 7036 107204 0 0 6 10 120 24 10 2 89
2 0 0 111304 9728 7036 107204 0 0 0 0 526 1653 96 4 0
1 0 0 111304 9616 7036 107204 0 0 0 0 552 2219 94 5 1
1 0 0 111304 9616 7036 107204 0 0 0 0 624 699 98 2 0
2 0 0 111304 9616 7052 107204 0 0 0 48 603 1466 95 5 0
3 0 0 111304 9620 7052 107204 0 0 0 0 768 932 90 4 6
3 0 0 111304 9440 7076 107360 92 0 244 0 820 1230 85 9 6
2 0 0 111304 9276 7076 107368 0 0 0 0 832 1060 87 6 7
3 0 0 111304 9624 7092 107372 0 0 16 0 813 1655 93 5 2
2 0 2 111304 9624 7108 107372 0 0 0 972 1189 1165 68 9 23
|
During this 10-second sample, the amount of free memory (the
free field) varies somewhat,
and there is a bit of swap-related I/O (the si and so fields), but overall this system is
running well. It is doubtful, however, how much additional workload
it could handle, given the current memory utilization.
When researching memory-related issues, it is often necessary to
determine how the Red Hat Enterprise Linux virtual memory subsystem
is making use of system memory. By using sar, it is possible to examine this aspect of system
performance in much more detail.
By reviewing the sar -r report, we can
examine memory and swap utilization more closely:
Linux 2.4.20-1.1931.2.231.2.10.ent (pigdog.example.com) 07/22/2003
12:00:01 AM kbmemfree kbmemused %memused kbmemshrd kbbuffers kbcached
12:10:00 AM 240468 1048252 81.34 0 133724 485772
12:20:00 AM 240508 1048212 81.34 0 134172 485600
…
08:40:00 PM 934132 354588 27.51 0 26080 185364
Average: 324346 964374 74.83 0 96072 467559
|
The kbmemfree and
kbmemused fields show the
typical free and used memory statistics, with the percentage of
memory used displayed in the %memused field. The kbbuffers and kbcached fields show how many kilobytes of
memory are allocated to buffers and the system-wide data cache.
The kbmemshrd field is
always zero for systems (such as Red Hat Enterprise Linux) using
the 2.4 Linux kernel.
The lines for this report have been truncated to fit on the
page. Here is the remainder of each line, with the timestamp added
to the left to make reading easier:
12:00:01 AM kbswpfree kbswpused %swpused
12:10:00 AM 522104 0 0.00
12:20:00 AM 522104 0 0.00
…
08:40:00 PM 522104 0 0.00
Average: 522104 0 0.00
|
For swap utilization, the kbswpfree and kbswpused fields show the amount of free
and used swap space, in kilobytes, with the %swpused field showing the swap space used
as a percentage.
To learn more about the swapping activity taking place, use the
sar -W report. Here is an example:
Linux 2.4.20-1.1931.2.231.2.10.entsmp (raptor.example.com) 07/22/2003
12:00:01 AM pswpin/s pswpout/s
12:10:01 AM 0.15 2.56
12:20:00 AM 0.00 0.00
…
03:30:01 PM 0.42 2.56
Average: 0.11 0.37
|
Here we notice that, on average, there were three times fewer
pages being brought in from swap (pswpin/s) as there were going out to swap
(pswpout/s).
To better understand how pages are being used, refer to the
sar -B report:
Linux 2.4.20-1.1931.2.231.2.10.entsmp (raptor.example.com) 07/22/2003
12:00:01 AM pgpgin/s pgpgout/s activepg inadtypg inaclnpg inatarpg
12:10:00 AM 0.03 8.61 195393 20654 30352 49279
12:20:00 AM 0.01 7.51 195385 20655 30336 49275
…
08:40:00 PM 0.00 7.79 71236 1371 6760 15873
Average: 201.54 201.54 169367 18999 35146 44702
|
Here we can determine how many blocks per second are paged in
from disk (pgpgin/s) and paged
out to disk (pgpgout/s). These
statistics serve as a barometer of overall virtual memory
activity.
However, more knowledge can be gained by examining the other
fields in this report. The Red Hat Enterprise Linux kernel marks
all pages as either active or inactive. As the names imply, active
pages are currently in use in some manner (as process or buffer
pages, for example), while inactive pages are not. This example
report shows that the list of active pages (the activepg field) averages approximately
660MB.
The remainder of the fields in this report concentrate on the
inactive list — pages that, for one reason or another, have
not recently been used. The inadtypg field shows how many inactive
pages are dirty (modified) and may need to
be written to disk. The inaclnpg field, on the other hand, shows
how many inactive pages are clean
(unmodified) and do not need to be written to disk.
The inatarpg field
represents the desired size of the inactive list. This value is
calculated by the Linux kernel and is sized such that the inactive
list remains large enough to act as a pool for page replacement
purposes.
For additional insight into page status (specifically, how often
pages change status), use the sar -R
report. Here is a sample report:
Linux 2.4.20-1.1931.2.231.2.10.entsmp (raptor.example.com) 07/22/2003
12:00:01 AM frmpg/s shmpg/s bufpg/s campg/s
12:10:00 AM -0.10 0.00 0.12 -0.07
12:20:00 AM 0.02 0.00 0.19 -0.07
…
08:50:01 PM -3.19 0.00 0.46 0.81
Average: 0.01 0.00 -0.00 -0.00
|
The statistics in this particular sar
report are unique, in that they may be positive, negative, or zero.
When positive, the value indicates the rate at which pages of this
type are increasing. When negative, the value indicates the rate at
which pages of this type are decreasing. A value of zero indicates
that pages of this type are neither increasing or decreasing.
In this example, the last sample shows slightly over three pages
per second being allocated from the list of free pages (the
frmpg/s field) and nearly 1
page per second added to the page cache (the campg/s field). The list of pages used as
buffers (the bufpg/s field)
gained approximately one page every two seconds, while the shared
memory page list (the shmpg/s
field) neither gained nor lost any pages.
