Memory Usage (What is using all of the memory?) - ITRC doc id MEMORYKBAN00000975
using all of the memory?) - ITRC doc id MEMORYKBAN00000975
Last modified: July 26, 2006
The
purpose of this
document is to describe how memory is used and the tools, both
supported and unsupported, that are availble to examine/report memory
usage. See below for details.
Monitoring Memory Usage
- There
are 3 ways for memory to be allocated, all requiring an
equivalent amount of swap.
- plain memory as allocated with malloc(3C) system call.
- shared memory as allocated with shmget(2)
system call. - memory mapped files as allocated with mmap(2) system call.
- plain memory
- Allocated by malloc(3C) system call.
- Can be examined with: ps, procsize, kmeminfo, and glance.
- Used by process’ data space, stack space, and text space
- Limited by kernel parameters:
- maxdsiz(5), maxssiz(5), and maxtsiz(5) respectively, for 32-bit apps
- maxdsiz_64bit(5), maxssiz_64bit(5), and maxtsiz_64bit(5) respectively, for 64-bit apps
- shared memory
- Allocated by shmget(2) system call.
- Can be examined with ipcs and with unsupported utilities called procsize, and shminfo.
- Limited by kernel parameter:
- memory mapped files
- NOTE: ps does NOT show shared
memory nor memory mapped files.
- For example:
- $ UNIX95= ps -eo vsz,ruser,pid,args | sort -rn | more
26332 ids 1685 ./idsagent -a
5296 root 2178 /usr/sbin/stm/uut/bin/tools/monitor/fpl_em
4760 root 2713 /opt/perf/bin/rep_server -t SCOPE /var/opt/perf/datafiles/loggl
4068 root 1487 /opt/perf/bin/scopeux
4052 root 1243 /opt/dce/sbin/rpcd
3364 root 2715 /opt/perf/bin/alarmgen -svr 2714 -t alarmgen /var/opt/perf/data
3180 root 1465 /opt/perf/bin/midaemon
3148 root 1495 /usr/sbin/swagentd -r - Look at the 1st column
in the output to
see the amount of memory used by this process for data/text and stack.
This value is in pages, so multiply by 4096 to determine the
size in bytes.
Anytime
you see that the size (SZ) is a four-digit number, that’s relatively large, so it’s one to watch over time
and to see if it continues to grow, and therefore may have a memory leak. - If you see a process called mib2agt using an excessive amount of
memory, this binary, mib2agt, has a known memory leak fixed
in an 11.X patch called PHSS_27858 (ITRC
ftp site download). This
patch DOES NOT require a reboot. Furthermore, mib2agt can be killed and then restarted with “kill mib2agt_PID”
and “/usr/sbin/mib2agt”. But only restart it if you have need of
supporting SNMP requests (e.g. OpenView). If not needed, can be
configured to not start at bootup by modifying /etc/rc.config.d/SnmpMib2.
- procsize breaks down memory by various types, specifically: UAREA, TEXT, DATA, STACK, Shared Memory, Memory Mapped files.
- For example:
$ ./procsize -fnc | head -3
pid
Comm UAREA
TEXT DATA STACK SHMEM
IO MMAP Total2916
getty
v 4
5 6
4 0
0 349 369
2287
prm3d
v 68
6 671
513 0
0 37212 38471$
- $ ./procsize -fcn |sort -rnk 11 | more
- $ ./procsize -fcn | sort -rnk 11 | head -50 >> /tmp/procsize.log
- Use the following to sort by MMAP column:
- $ ./procsize -fcn |sort -rnk 10 | more
- For example:
- $ ./kmeminfo –user
kmeminfo
(3.57)
libp4
(7.124): Opening /stand/vmunix /dev/kmem
Boot time: Mon Nov 25 12:01:58 2002
Dump
time: Mon Jan 6 12:24:21 2003———————————————————–Summary
of user processes memory usage:Process
list sorted by resident set size …
proc vas
p_pid va_rss va_prss va_ucount command0x0ab6180
0x1de3400
3185 3895
3865 7678 mxagent0x0abca00
0x1c1b900
1538 2051
2040 8867 rbootd0x0ac19c0
0x1f46f00
3184 1563
1533 7207 mxrmi0x0abccc0
0x1d55800
2434 1236
1032 5364 rds0x0acb3c0
0x2095200 12454
919
889 7415 kmeminfo0x0ac2d00
0x1e49800
2635 527
471 8715 ns-slapd0x0ac1f40
0x1df5a00
2684 380
365 3371 ns-admin0x0abc480
0x1bbb000
1476 359
277 3349 dmisp0x0ac4300
0x1e83600
2853 318
264 4237 hpwebjetd
- A trial
version of Glance is available on
the Application Media. - Glance is not available
for download.
- For example:
- Process List
- PROCESS LIST
Users= 5
User CPU Util
Cum
Disk Thd
Process
Name PID PPID Pri Name ( 100%
max) CPU IO Rate RSS Cnt——————————————————————————–
pax
13819 13818 148 root 2.7/
5.8 273.3 9.4/32.8
284kb 1glance
14464 1822 158 root 2.1/
3.1 3.0 0.0/ 2.1
4.3mb 1scopeux
1715 1 127
root 1.7/ 0.2 518.4
1.5/ 0.0 4.1mb 1swapper
0 0 127
root 1.5/ 0.8 2213.0 0.3/
0.0 16kb 1java
10095 1 168
root 1.0/ 2.7 348.7
0.0/ 4.2 42.0mb 28vxfsd
35 0 138
root 0.2/ 0.1 289.4
1.9/ 1.3 352kb 16
- Application List
-
APPLICATION LIST
Users= 5
Num Active CPU AvgCPU Logl
Phys Res VirtIdx
Application Procs
Procs Util Util
IO IO Mem Mem——————————————————————————–
1
other
2 0 0.0
0.0 0.0 0.0 804kb
19.3mb
2
network
55 5 0.4
0.3 0.0 0.0 12.1mb
35.4mb
3 memory_management
3 3 1.6
1.8 0.0 1.1
96kb 376kb
4 other_user_root
101 34 52.3 43.1
60.9 65.2 109.9mb 614.0mb
- In the output of
shminfo, memory mapped files are shown as “OTHER”. - For example:
- $ ./shminfo
Shared space from Window id 0 (global):
Space Start End Kbytes Usage
Q2 0x00006fea.0x40000000-0x7fff0000 1048512 FREE
Q3 0x00000000.0x80000000-0x80001000 4 SHMEM id=0
Q3 0x00000000.0x80001000-0x80002000 4 OTHER
Q3 0x00000000.0x80002000-0x80102000 1024 SHMEM id=201
Q3 0x00000000.0x80102000-0x81202000 17408 OTHER
Q3 0x00000000.0x81202000-0x8121b000 100 SHMEM id=3602
Q3 0x00000000.0x8121b000-0x81eea000 13116 FREE
Q3 0x00000000.0x81eea000-0x81efd000 76 SHMEM id=3
Q3 0x00000000.0x81efd000-0x82df0000 15308 OTHER
Q3 0x00000000.0x82df0000-0x82df6000 24 SHMEM id=4004
Q3 0x00000000.0x82df6000-0x83aa6000 12992 OTHER
- For example:
- $ ipcs
–mpb | more
IPC status from /dev/kmem as of Wed Mar 3 07:39:51 2004
T
ID
KEY
MODE
OWNER GROUP SEGSZ CPID LPIDShared Memory:
m
0 0x41200007 –rw-rw-rw-
root root
348 636 636m
1 0x4e000002 –rw-rw-rw-
root root 61760
636 638m
2 0x41241878 –rw-rw-rw-
root root 8192
636 638m
3 0x000024ef –rw-rw-rw-
root root 7712
1143 1137m
4 0x30205f0d –rw-rw-rw-
root root 1048576 1184 1226m 1605
0x0c6629c9 –rw-r—–
root root 19059552 1823 13457m
606 0x49180013 –rw-r–r–
root root 22908 1804
1903m
7 0x06347849 –rw-rw-rw-
root root 77384 1823
1903m 7208
0x5e1c019c –rw——-
root sys 512
19627 19627m 3409
0x00000000 D-rw——-
root root 213272 2198 2198m
10 0x011c0082 –rw——-
www other 100000 2203 2204
usage, run:
- $ ipcs -mpb | sed
-n ‘/^m/p’ | awk ‘{total+=$(NF-2)}END{printf(“%d\n”, total)}’
- And if total is at or near 1.75 Gb or 2.75Gb
then address as a 32-bit
limitation issue.
- To find processes, if still running, that last touched (LPID) shared memory segments:
- $ ps -ef | `ipcs -mpb | sed -n ‘/^m/p’ | \
awk ‘{printf(”%s “, $NF)} END{printf(”\n”)}’ | \
sed ’s/\ /\|/g’| sed ’s/\|$//’ | \
awk ‘{printf(”egrep -e %s\n”,$0)}’ | \
sed ’s/ \-e / \-e \”/’ |sed ’s/$/\”/’`
Blogged with Flock
