Memory Usage (What is using all of the memory?) - ITRC doc id MEMORYKBAN00000975

July 26, 2006
Memory Usage (What is
using all of the memory?) - ITRC doc id MEMORYKBAN00000975

by:  eric.herberholz@hp.com

Last modified: July 26,  2006

Full document is available at external ftp site: ftp://eh:spear9@hprc.external.hp.com/memory.htm

The “Table of Contents” is available as ITRC doc id MEMORYKBAN00000975

The
purpose of this
document is to describe how memory is used and the tools, both
supported and unsupported, that are availble to examine/report memory
usage.  See below for details.

Monitoring Memory Usage

  • There
    are 3 ways for memory to be allocated, all requiring an
    equivalent amount of swap.
    1. plain memory as allocated with  malloc(3C) system call.
    2. shared memory as allocated with shmget(2)
      system call.
    3. memory mapped files as allocated with mmap(2) system call. 


    1. plain memory

    2. shared memory 
    3. memory mapped files

    • NOTE: ps does NOT show shared
      memory nor memory mapped files.

    • For example:

    • procsize breaks down memory by  various types, specifically: UAREA, TEXT, DATA, STACK, Shared  Memory, Memory Mapped files.
    • For example:

      $ ./procsize -fnc | head -3


      pid
      Comm        UAREA  
      TEXT   DATA  STACK  SHMEM    
      IO  MMAP    Total

      2916
      getty   
      v    4     
      5      6      
      4     0     
      0    349      369


      2287
      prm3d   
      v   68     
      6     671   
      513     0     
      0  37212    38471

      $

  1. Here’s the command to use to sort  by the 11th column (total memory usage)
    • $ ./procsize -fcn |sort -rnk 11 | more
  2. To save top 50 memory users in a file:
    • $ ./procsize -fcn | sort -rnk 11 | head -50 >> /tmp/procsize.log
  3. In the output of procsize, shared memory usage is shown under the “SHMEM” column.
  4. In the output of procsize, memory mapped  files are shown under the “MMAP” column.
    • Use the following to sort by MMAP column: 
    • $ ./procsize -fcn |sort -rnk 10 | more

    • For example:
    • $ ./kmeminfo –user
      kmeminfo
      (3.57)
      libp4
      (7.124): Opening /stand/vmunix /dev/kmem
      Boot time: Mon Nov 25 12:01:58 2002
      Dump
      time: Mon Jan  6 12:24:21 2003

      ———————————————————–

       Summary
      of user processes memory usage:


      Process
      list sorted by resident set size …



           proc      vas  
      p_pid   va_rss  va_prss va_ucount command

      0x0ab6180
      0x1de3400 
      3185     3895     
      3865     7678 mxagent

      0x0abca00
      0x1c1b900 
      1538     2051     
      2040     8867 rbootd

      0x0ac19c0
      0x1f46f00 
      3184     1563     
      1533     7207 mxrmi

      0x0abccc0
      0x1d55800 
      2434     1236     
      1032     5364 rds

      0x0acb3c0
      0x2095200 12454     
      919      
      889     7415 kmeminfo

      0x0ac2d00
      0x1e49800 
      2635      527      
      471     8715 ns-slapd

      0x0ac1f40
      0x1df5a00 
      2684      380      
      365     3371 ns-admin

      0x0abc480
      0x1bbb000 
      1476      359      
      277     3349 dmisp

      0x0ac4300
      0x1e83600 
      2853      318      
      264     4237 hpwebjetd

    • A trial
      version of Glance is available on
      the Application Media.

    • Glance is not available
      for download.

    • For example:
    • Process List
    •                                  PROCESS LIST                     
      Users=    5

                                   
      User      CPU Util    
      Cum    
      Disk           Thd

      Process
      Name   PID   PPID Pri Name   (  100%
      max)    CPU   IO Rate    RSS   Cnt

      ——————————————————————————–

      pax         
      13819  13818 148 root      2.7/
      5.8   273.3  9.4/32.8  
      284kb    1

      glance      
      14464   1822 158 root      2.1/
      3.1     3.0  0.0/ 2.1  
      4.3mb    1

      scopeux      
      1715      1 127
      root      1.7/ 0.2   518.4 
      1.5/ 0.0   4.1mb    1

      swapper         
      0      0 127
      root      1.5/ 0.8  2213.0  0.3/
      0.0    16kb    1

      java        
      10095      1 168
      root      1.0/ 2.7   348.7 
      0.0/ 4.2  42.0mb   28

      vxfsd          
      35      0 138
      root      0.2/ 0.1   289.4 
      1.9/ 1.3   352kb   16

    • Application List
    •                                
      APPLICATION LIST                   
      Users=    5

                               
      Num Active  CPU  AvgCPU  Logl  
      Phys     Res    Virt

      Idx
      Application         Procs 
      Procs  Util  Util   
      IO     IO      Mem     Mem

      ——————————————————————————–

       
      1
      other                   
      2     0   0.0  
      0.0    0.0    0.0   804kb 
      19.3mb

       
      2
      network                
      55     5   0.4  
      0.3    0.0    0.0  12.1mb 
      35.4mb

       
      3 memory_management       
      3     3   1.6  
      1.8    0.0    1.1   
      96kb   376kb

       
      4 other_user_root       
      101    34  52.3  43.1  
      60.9   65.2 109.9mb 614.0mb

    • In the output of
      shminfo, memory mapped  files are shown as “OTHER”.
    • For example:
    • $ ./shminfo

      Shared space from Window id 0 (global):

      Space      Start       End  Kbytes Usage

      Q2 0x00006fea.0x40000000-0x7fff0000 1048512 FREE

      Q3 0x00000000.0x80000000-0x80001000      4 SHMEM id=0

      Q3 0x00000000.0x80001000-0x80002000      4 OTHER

      Q3 0x00000000.0x80002000-0x80102000    1024 SHMEM id=201

      Q3 0x00000000.0x80102000-0x81202000   17408 OTHER

      Q3 0x00000000.0x81202000-0x8121b000     100 SHMEM id=3602

      Q3 0x00000000.0x8121b000-0x81eea000   13116 FREE

      Q3 0x00000000.0x81eea000-0x81efd000     76 SHMEM id=3

      Q3 0x00000000.0x81efd000-0x82df0000   15308 OTHER

      Q3 0x00000000.0x82df0000-0x82df6000     24 SHMEM id=4004

      Q3 0x00000000.0x82df6000-0x83aa6000   12992 OTHER

    • For example:

    • $ ipcs
      –mpb  | more

      IPC status from /dev/kmem as of Wed Mar  3 07:39:51 2004

      T     
      ID    
      KEY       
      MODE       
      OWNER     GROUP  SEGSZ  CPID  LPID

      Shared Memory:

      m      
      0 0x41200007 –rw-rw-rw-     
      root      root   
      348   636   636

      m      
      1 0x4e000002 –rw-rw-rw-     
      root      root  61760  
      636   638

      m      
      2 0x41241878 –rw-rw-rw-     
      root      root   8192  
      636   638

      m      
      3 0x000024ef –rw-rw-rw-     
      root      root   7712 
      1143  1137

      m      
      4 0x30205f0d –rw-rw-rw-     
      root      root 1048576  1184  1226

      m    1605
      0x0c6629c9 –rw-r—–     
      root      root 19059552  1823 13457

      m    
      606 0x49180013 –rw-r–r–     
      root      root  22908  1804 
      1903

      m      
      7 0x06347849 –rw-rw-rw-     
      root      root  77384  1823 
      1903

      m    7208
      0x5e1c019c –rw——-     
      root       sys    512
      19627 19627

      m    3409
      0x00000000 D-rw——-     
      root      root 213272  2198  2198

      m     
      10 0x011c0082 –rw——-      
      www     other 100000  2203  2204

  5. To total the shared memory
    usage, run: 
    • $ ipcs -mpb | sed
      -n ‘/^m/p’ |
      awk ‘{total+=$(NF-2)}END{printf(“%d\n”, total)}’

    • To find processes, if still running, that last touched (LPID) shared memory segments:
    • $ ps -ef | `ipcs -mpb | sed -n ‘/^m/p’ | \

          awk ‘{printf(”%s “, $NF)} END{printf(”\n”)}’ | \

          sed ’s/\ /\|/g’| sed ’s/\|$//’ | \

          awk ‘{printf(”egrep -e %s\n”,$0)}’ | \

          sed ’s/ \-e / \-e \”/’ |sed ’s/$/\”/’`

    Blogged with Flock

    nflocks

    nflocks(5) - number of file locks

    • References
    • Check /var/adm/syslog/syslog.log
    • it is the application’s responsibility to write to syslog regarding inability to obtain a file lock
    • for example:
      Jun 24 17:13:11 usnbru25 smbd[5407]:
         nt_printing_init: Failed to open nt formsdatabase /var/opt/samba/locks/ntforms.tdb (No locks available)
    Dec 4 10:06:15 frodo nmbd[1243]: tdb(/usr/local/samba/locks/unexpected.tdb): tdb_lock failed on list 3 ltype=2 (No locks available)
    Dec 4 10:05:38 frodo qpopper[27188]: csmith at 192.168.0.152 (192.168.0.152): -ERR [SYS/TEMP] flock: ‘/var/mail/.csmith.pop’: No locks available (46)

    • Monitoring file lock usage - with Glance, with q4 script, or with lsof

    • Glance shows “File Locks (nflocks)” usage in the System Tables screen. (keyboard short cut is ‘t’). To monitor usage, here’s a glance adviser script below.  This script can be run periodically from cron to monitor nflocks usage over time.
    • nflocks.adviser script:
      print GBL_STATDATE,” “,GBL_STATTIME,” Total
      nflocks: “,TBL_FILE_LOCK_AVAIL,”  nflocks in-use: “,TBL_FILE_LOCK_USED
    • command line to run from
      cron
      :
      # glance -j 1 -iterations 1 -adviser_only
      -syntax \

      /home/erich/nflocks.adviser 2>/dev/null | head -1
      >>/var/adm/nflocks.log
    • q4 script -
      #!/usr/bin/sh

      /usr/contrib/bin/q4
      /stand/vmunix /dev/mem
        <<-EOF
      load struct
      locklist from &locklist max nflocks

      keep ll_count !=
      0

      EOF


      (note: You may need to run
      q4pxdb
      /stand/vmunix
      ” if you get complaints
      from q4)

    • For example:

      # ./nflocks.sh

      see help topic q4live regarding use of /dev/mem

      Reading kernel symbols …

      Reading data types …

      Initialized PA-RISC 1.1 (no buddies) address translator …

      Initializing stack tracer …

      Get the latest Q4 news by typing “news | more”.

      loaded 200 struct locklists as an array (stopped by max count)

      kept 7 of 200 struct locklist’s, discarded 193

      #

    • lsof - The unsupported tool called lsof can be used.
    • For example:
    • For a pure count of how many locks are in-use, use:
    • # ./lsof | awk ‘{print $4}’| sed -n ‘/^[0-9]/p’ |sed ’s/[0-9]*//’ | awk ‘{if (length($1) > 1) print $1}’ | wc -l
    • NOTE:this magic brought to you by carefull reading of the lsof(8) man pagethat explains the FD (4th field) as a file descriptor number followedby rwu, space, or “-”, and then followed by one of NrRwWuUx (see lsof(8) man page at “The HP-UX Porting and Archive Centre”:
      [Canada  UnitedKingdom  USA Utah] for full details).

  6. NOTE: lsof is an unsupported utility available
  7. Tips - Searching Tip: use site feature, e.g site:hp.com

    July 22, 2006

    For example,

    1. Searchig for CommandView 1.08 for 11.11 at hp’s sofware depot www.hp.com/go/sofware.com turned up nothing.
    2. Searching at HP’s Business Support Center, turned up a version for 11.00 and 11.23, but not for 11.11.
    3. Someone pointed out that for public searches, use Google and someone pointed out to use the “site” feature, so I searched for "site:hp.com  download  commandview  11.11" using my favorite search engine, Dogpile*  … And it worked very well!!!
    * Dogpile searches Google, Yahoo!Search, MSN*Search and Ask.

    Blogged with Flock

    Lemon Garlic Chicken

    July 5, 2006

    Combine in pot:
    * 1/2 c. butter
    * 1 c. lemon juice
    * 4 cloves of garlic
    * 1 t. black pepper

    Heat through, but don’t boil.

    Don’t need to mariante, simply baste 2 chickens throughout grilling. Half chickens cooked with ribs down won’t burn skin nor meat.

    Blogged with Flock

    Chinese Cabbage Salad

    July 4, 2006

    Combine in bowl:
    * 1 head cabbage, chopped
    * 2 packs top ramen noodes, broken into pieces (without flavor packets)
    * 1 C. chicken (optional)
    * 1/2-3/4 C. snow peas (optional)
    * 8 green oninons, chopped (optional)

    Heat in pan until slightly brown:
    * 2 T. butter
    * 1/3 C. almonds, sliced or slivered
    * 1/3 C. sesame seeds

    Add to bowl.

    Make dressing - combine and shake:
    * 1 t black pepper
    * 1/4 C. sugar
    * 2/3 C. oil
    * 1 1/3 C. rice vinegar

    Add Dressing to bowl, stir, and refigerate until ready to serve/eat.

    Get free blog up and running in minutes with Blogsome
    Theme designed by Viewfinder Design