Introduction

[ TOP ]

• The memory
  line is
  infamously misleading and does not refer to acual physical memory use!!!! Rather it is
  the size of pseudoswap, which happens to be
  calculated to be 75% of the size of RAM (a.k.a. memory.)  As the name (”pseudo”)
  implies, it does NOT exist. Pseudoswap is enabled by default
  with the kernel parameter swapmem_on(5) set to 1.  Don’t worry
  about this line, just look at the total line for total used and
  sometimes it’s interesting to look at the device PCT USED as an
  indication of how much swapping has occurred since the box was last
  rebooted. The memory
  line is also described  in the swapinfo(1m) man page.

• To read more
  on pseudoswap, refer to:

• Pseudo-Swap Space section of the Memory Management white paper
• “What is swap and why
  does a system need it?“   ITRC doc id  KBRC00001262
• “SYSADM: Swap and Pseudoswap Clarified“  ITRC Doc id  KBRC00010136

  
  

[ TOP ]

swapinfo description and example…

• dev line(s):

• memory line:

• when present, indicates  pseudo-swap is enabled   (i.e.  swapmem_on kernel paraemter is set to 1, which  is
  the default.) The size of pseudoswap is calculated to be 75% of the size of RAM (a.k.a. memory.)  In other words, it does not refer to acual physical memory use!!!!  Pseudo-swap was designed specifically for large memory
  systems for which acutal
  swapping is never (or rarely) expected to occur, so there’s less need
  to
  use actual
  physical disk space for
  swap. For more information, see swapmem_on(5)
  , which reads:

  
  

         In
  previous versions of HP-UX, system configuration required sufficient
  physical swap space for the maximum possible
  
  
  
         number
  of processes on the system. This is because HP-UX reserves swap space
  for a process when it is created, to
  
  
  
         ensure
  that a running process never needs to be killed due to insufficient
  swap.
  
  
        
  This
  was difficult, however, for systems needing gigabytes of swap space
  with gigabytes of physical memory, and those
  
  
  
         with
  workloads where the entire load would always be in core. This tunable
  was created to allow system swap space to
  
  
  
         be
  less than core  memory. To accomplish this,
  a portion of physical memory is set aside as ‘pseudo-swap’ space.
  
  
         While
  actual swap space is still available, processes still reserve all the
  swap they will need at fork or execute time from
  
  
  
         the
  physical device or file system swap. Once this swap is completely used,
  new processes do not reserve swap, and
  
  
         each
  page which would have been swapped to the physical device or file
  system is instead locked in memory and
  
  
         counted
  as part of the pseudo-swap space.

• total
  line:

[ TOP ]

• The Buffer Cache size is, by default, 50% of RAM (see kernel parameter dbc_max_pct(5)). 

• Some consider there to be a buffer cache
  sweet spot -  400 Mb or 20% of memory, whichever is
  smaller. But of course, this may
  vary depending on the applications/dbs used on the system. 

• To check the current size of the
  buffer cache, either “sysdef | grep bufpages” (and multiply by 4096
  to approximate the current size of buffer cache) or use glance’s memory
  screen to see what size “BufCache” is.

•   Note: Although buffer cache can be dynamic in size, decrease only occurs under memory pressure and then
  only decreases very slowly. So, the buffer cache often grows
  farily quickly to dbc_max_pct and only decreases (and
  slowly) when memory presssure is high.

Q: How much RAM(memory)
does system have?

   
      a.    11.23        # echo phys_mem_pages/2d | adb /stand/vmunix /dev/kmem

                                
11.11   # echo phys_mem_pages/D | adb -k
/stand/vmunix /dev/mem

             10.x    # echo physmem/D | adb -k /stand/vmunix
/dev/mem

                   
     b. multiply output of adb by 4096 to get the size of RAM.

    
            2. Run:

           
# dmesg | grep Phys

   
             3. Check with glance in the Memory Report, look at the value of Phys Mem.

[ TOP ]

B. Monitoring Memory Usage

1. plain memory as allocated with  malloc(3C) system call.
2. shared memory as allocated with shmget(2)
   system call.
3. memory mapped files as allocated with mmap(2) system call.  

[ TOP ]             

1. plain memory
   • as allocated with malloc(3C), can be examined with: ps, procsize, kmeminfo, and glance.  See below for details on each of these utilities/tools.
   • is used by process’ data space, stack space, and text space, limited by kernel parameters  maxdsiz(5), maxssiz(5), and maxtsiz(5) respectively, for 32-bit apps; and maxdsiz_64bit(5), maxssiz_64bit(5), and maxtsiz_64bit(5) respectively, for 64-bit apps.

   
   

• Alternatively, to look at both virtual size as well as
  the actual size:

• procsize 
  is available for
  download here:
  
    System:     hprc.external.hp.com  (192.170.19.51)
  

    Login:      eh
  
    Password:   spear9
  
    ftp://eh:spear9@hprc.external.hp.com/procsize.README.txt

    ftp://eh:spear9@hprc.external.hp.com/procsize.64bit.sh

    ftp://eh:spear9@hprc.external.hp.com/procsize.32bit.sh

• For example:

• Look at breakdown of memory usage, per process:

  # ./procsize -fnc | more
  
  pid
  Comm        UAREA  
  TEXT   DATA  STACK  SHMEM    
  IO  MMAP    Total

  
  2916
  getty   
  v    4     
  5     6     
  4     0     
  0      349      369

  2287
  prm3d   
  v   68     
  6   671   
  513     0     
  0    37212    38471

  .

  .

  .

• NOTE: numbers in the output of  procsize are the number of 4K pages. So, mulitply by 4096 to get a byte count.

• TRICK:  Here’s the command to use to sort  by the 11th column (total memory usage)

                c) Can also look at memory
usage, by process, with an unsupported utility called kmeminfo. 

• kmeminfo
  is an unsupported utility
  that can be used to examine/report memory usage.   
  
  kmeminfo
  is available for download here:

    System:     hprc.external.hp.com  (192.170.19.51)
  

    Login:      eh
  
    Password:   spear9

    ftp://eh:spear9@hprc.external.hp.com/kmeminfo.README.txt

    ftp://eh:spear9@hprc.external.hp.com/kmeminfo.64bit.sh

    ftp://eh:spear9@hprc.external.hp.com/kmeminfo.32bit.sh

• For example:

            
# ./kmeminfo –user

                 
d) Can use glance’s process list or application list. For example:

• A trial
  version of Glance is available on
  the application CDs (usally on cd #2 or #3)

• Glance is not available
  for download.

  Glance  product
  #’s

• 11.x s700:  
  B3691AA  B3699AA

• Trial version: B3691AA_TRY   B3699AA _TRY

• Trial version: B3693AA_TRY B3701AA_TRY

[ TOP ]

• 2) shared memory as allocated with shmget(2) system call.

• a) Can look at shared memory usage with  ipcs(1).  For example:

• b) Can look at
  shared memory usage with an unsupported  tool called shminfo

• shminfo
  is available for download here:

    System:     hprc.external.hp.com  (192.170.19.51)
  

    Login:      eh
  
    Password:   spear9

 
                                 ftp://eh:spear9@hprc.external.hp.com/shminfo.README.txt

              ftp://eh:spear9@hprc.external.hp.com/shminfo.64bit.sh

              ftp://eh:spear9@hprc.external.hp.com/shminfo.32bit.sh

• For example:

             
# ./shminfo

             
# ./shminfo –64bit

• TRICK: 

• If a particular shared
  memory segment is of interest, and if you want to know which processes
  are attached to that shared memory segment, you can use shminfo -s id (where id is the shared memory identifier.)

• For example:

• c) Can look at shared memory
  usage by process with an unsupported utility called procsize, which breaks down memory by  various types, specifically: UAREA, TEXT, DATA, STACK, Shared  Memory, Memory Mapped files.

          [ TOP ]

• 3)
  memory
  mapped files as allocated with mmap(2) system call. 

• a) The are no system
  commands that will report memory mapped file usage

• b) Can use an unsupported utility called shminfo to see use of memory mapped memory.

• In the output of
  shminfo, memory mapped  files are shown as “OTHER”.

• See above
  for examples and download site for shminfo.

• In the output of procsize , memory mapped  files are shown under the “MMAP” column.

• See above
  for examples and download site for procsize

• TRICK: Use the following to sort by MMAP column:

     # ./procsize -fcn |sort -rnk 10 | more

[ TOP ]

• 1. Check for known OS Memory Leaks with an unsupported utility
  called kmeminfo The Response Center can
  assist with using the output of kmeminfo.

• The JFS inode cache, is sized by the vx_ninode kernel parameter.  The
  default value of vx_ninode is determined by size of RAM and is
  different for 11.00 -vs- 11.11. 

[ TOP ]

1. Use a tool to capture a baseline of memory use per process,
2. Then gather subsequent
   reports to see if there is a steady increase in
   memory use.

• it is out of swap
• it hit the maxdsiz limit (a kernel tunable parameter)

• it is out of addressible data space, e.g. max for 32-bit apps is ~940 Mb or ~1.9 Gb (for more details, see here)

    3. Check with vendor of application (e.g. for known issue/patch).

    4. Or to check yourself, try Purify^TM or Insure++^TM to troubleshoot.

 
NOTE: these are not endorsements:

        Purify^TM  is availble at http://www.rational.com

        Insure++^TM is availble at http://www.parasoft.com
    

[ TOP ]

• Running without memory windows, HP-UX has limitations for shared resources on
  32-bit applications. All applications in the system share and are limited to a total of
  1.75GB of shared memory, 2.75GB if compiled as SHMEM_MAGIC. In a system with
  16GB of physical memory, only 1.75 can be used for shared resources!!!! To
  address this limitation, a functional change has been made (Memory Windows
  was introduced by patches at 11.0 and is included in 11.11) to allow 32-bit
  processes to create unique memory windows for shared objects like shared
  memory.

• For 32
  bit applications, the
  maximum size of a single shared memory segment a process can attach is
  limited to 1 GB (shmmax
  <= 0x40000000). The first 1 Gb resides in quadrant 3, and quadrant 4
  only has 0.75 Gb reserved for shared memory and memory
  mapped files. So the total memory addressable by the default executable
  type is 1.75 Gb. 

  An executable type SHMEM_MAGIC has been defined
  which adds the use of quadrant 2 for shared memory and memory mapped
  files.  This additional 1 Gb results in a
  system-wide maximum of 2.75 GB of shared memory ­and
  memory mapped files address space. See below
  for more details about SHMEM_MAGIC. (note: for more on 32-bit memory limitation, see:
  “Understanding Shared Memory on PA-RISC Systems”,
  ITRC Doc id RCMEMKBAN00000027.)

• If you see “out of memory” or “not enough space” when running an application, and there is pleny of free swap space then the
  application may be requesting shared memory or may be mapping files to memory (with shmget() and mmap() system calls respectively) and the problem may due
  to 32-bit memory limitation/contention and the options are…

• Q: Is application 32-bit or 64-bit?

• A: To determine if an application binary
  is  32-bit or 64-bit, use the file(1)

• 32-bit example:

[ TOP ]

• SHMEM_MAGIC executables can address 2.75 Gb
  • For more details than
    explained below (about SHARE_MAGIC, EXEC_MAGIC, AND SHMEM_MAGIC),
    see ITRC doc id rcfaxmemory001
    (”Shared_magic
    Explained“)

• SUMMARY
  • To get SHMEM_MAGIC, the executable needs to
    have been previously linked with
    EXEC_MAGIC (”ld -N“) and then can be chatr‘ed to get SHMEM_MAGIC (i.e. with the
    “chatr -M” option.)

  • Check with the vendor’s
    application support to see if their application supports SHMEM_MAGIC.

  • For example, for Sybase
    support of SHMEM_MAGIC, see: http://my.sybase.com/detail?id=1010341

• PATCHES - Unlike 11.0 which doesn’t require
  patches for SHMEM_MAGIC, 10.20
  needs patches. The 10.20
  patches are:
  • PHKL_16750
    (for s700) PHKL_16751
    (for s800)
    • These are
      LITS (Line-In-The-Sand) patches and will never be superseded.

  • PHSS_21110
    (linker/ld
    patch)
    • Note:
      PHSS_21110,
      may be superseded. Please check for
      the latest patches at the IT ResourceCenter
      (ITRC) at the following web site:      
      http://www.itrc.hp.com/

• Q: Is
  application set up with SHMEM_MAGIC?

• A: To determine if
  the 32-bit application is setup with SHMEM_MAGIC
  or is capable of SHMEM_MAGIC, use the chatr(1) command. For example:

[ TOP ]

• Normally, 32bit
  apps can, as
  allowed by maxdsiz, can get up to ~940 Mb of data space, unless
  they have been linked with EXEC_MAGIC, in which case, they can get upwards of 1.9 Gb of data space.

• Alternatively, 
  chatr(1)
  may be
  used to enable the third quadrant
  private for data space.  Thus enabling an addition 1 Gb quadrant to be used for data space.

• For example:    
  chatr +q3p enable executable_name

• if it shows
  as “shared
  executable“  then it can only get to about 940Mb,
• if it shows “EXEC_MAGIC“,
  then it can get upwards of 1.9 Gb of data space.

• maxdsiz  - maximum size (in bytes) of the data segment/space for any user
  process.

[ TOP ]

• Overview - Running without memory
  windows, HP-UX has limitations for shared resources on 32-bit
  applications. All applications in the system are limited to a total of
  1.75GB of shared memory, 2.75GB if compiled as SHMEM_MAGIC. In a system
  with 16GB of physical memory, only 1.75 can be used for shared
  resources!!!! To address this limitation, a functional change has been
  made (Memory Windows was introduced by patches
  at 11.0) to allow 32-bit processes to create unique memory
  windows for shared objects like shared memory.  This allows
  cooperating applications to create 1GB of shared resources without
  exhausting the system-wide resource. Part of the virtual address space
  remains globally visible to all processes, so that shared libraries are
  accessible no matter what memory window they are in.  The
  following customer-visible changes have been made for memory windows:
  
  • New kernel tunable, max_mem_window(5), to configure the number of memory
    windows a system can have.
  • New set of commands
    and files and their assciated man pages:
    • getmemwindow(1M)
    • setmemwindow(1M)
    • services.window(4)

  
  

• There is a Memory Windows White Paper in pdf
  format and is also availabine in an ascii text version -  /usr/share/doc/mem_wndws.txt

• There is a Memory Windows
  summary in the 11i release notes.

• Memory Windows are NOT supported by 

• OmniBack

• see “/usr/lib/dld.sl: Call to mmap() failed - TEXT /opt/omni/gui/bin/libcore.sl” ITRC doc id OV-EN007127

• OpenView
  (see non-support statement for HP-UX 11 Memory Windows in the OVO/UNIX 7.1 Release Notes.)

• Oracle 8.06, or later “Using memory windows with
  Oracle” ITRC Doc Id: KBRC00009528

• GOTCHA: The default (SHARE_MAGIC) executable’s
  maximum size memory window is 1 gigabyte. Any consumption beyond 1
  gigabyte consumes space from the 4th quadrant which is shared across *ALL* processes in the system. This is important,
  any application within a memory window that uses more than 1 gigabyte
  of shared memory consumes quadrant 4 resources that are shared by all
  processes no matter what memory window they occupy.

• Yet More Details…

• Check with the
  vendor’s application support to see if they support Memory Windows.

• Patches - Memory Windows was originally introduced with 11.0 Patches: 

       PHKL_13810 (Kernel)

             PHCO_13811 (Commands)

• * check the
  ITRC http://itrc.hp.com for the latest versions of patches.

• 11.11 (11i) does not need
  patches for Memory Windows.

• Is system already configured for Memory Windows?

• Either, test with  setmemwindow(1M),
  for example:

      # setmemwindow date

• Memory Windows is not configured if you get
  nothing from the date(1) command (i.e. nothing comes back), or if you get an error
  like this:

      Error(12), unable to set memory
  window(-1)
  

• Or check the kernel parameter max_mem_window to see if it has been set, with:

      # grep max_mem_window /stand/system

• NOTE: If you do NOT see max_mem_window(5) as a kernel configurable parameter in SAM, then you can install
  the latest 11.0 SAM patch  (SAM was first made aware of max_mem_window with PHCO_21187) or you can add
  max_mem_window to system file manually and then generate
  a new kernel.

• memwin_stat: 

• is an unsupported utility that displays
  information about shared  memory segments, processes and the Memory Windows
  themselves.
• is available for download, here:

    System:     hprc.external.hp.com  (192.170.19.51)
  

    Login:      contrib
  
    Password:   9unsupp8

        ftp://contrib:9unsupp8@hprc.external.hp.com/sysadmin/memwin/memwin_stats

• and for download, here: 

    System:     hprc.external.hp.com  (192.170.19.51)

    Login:      eh

    Password:   spear9

    URLs:       

      ftp://eh:spear9@hprc.external.hp.com/memwin_stats.README.txt

      ftp://eh:spear9@hprc.external.hp.com/memwin_stats1100.sh

      ftp://eh:spear9@hprc.external.hp.com/memwin_stats1111.sh

      ftp://eh:spear9@hprc.external.hp.com/memwin_stats.html

• Footnote
  about memwin_stats availability:

• The PDF version of the Memory Windows White Paper (http://docs.hp.com/hpux/onlinedocs/os/memwn1_4.pdf)
  points the reader to an ftp site: ftp://contrib:9unsupp8@hprc.external.hp.com/sysadmin/memwin/memwin_stats
• The text version of the Memory Windows White Paper
  (/usr/share/doc/mem_wndws.txt) points
  to an ftp site:

     ftp://hpchs.cup.hp.com/tools/11.X/memwin_stats.shar

[ TOP ]

[ TOP ]

• Physical Memory
  • Can use dmesg to report Physical memory (RAM) info. For example:

       # dmesg |
    grep Phys
    

       Physical: 212992
    Kbytes, lockable: 152792 Kbytes, available: 178188 Kbytes

  • Note:
  • This system
    has 2GB physical memory.
  • Lockable memory is used for
    • Process
      images and overhead locked using the plock() system call (see HP-UX
      Reference entry plock(2)).
    • Shared
      memory segments locked with the SHM_LOC command of the shmctl() system
      call (see HP-UX Reference entry shmctl(2)).
    • Miscellaneous
      dynamic kernel data structures used by the shared memory system and
      some drivers

[ TOP ]