Linux ulimit: All about Process Stack Size
In many application installation, e.g. SAP BusinessObject, Oracle RDBMS, IBM Cognos Analytics, there is always a section about ulimit configuration like below:$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 127458
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 4096
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 2047
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
Stack is the maximum allowable size, in KB, that this specific UNIX user allow for each of the UNIX process. If it is multi-threaded, then all threads going to share the same stack size, In other word, the stack size will be total of all threads consumed. Default, RedHat/CentOS/Fedora set it to 10 MB.
For stack usage, 10MB is consider very high, as it is a memory area that used to perform stack push and pop (research assembly language PUSH and POP for more detail) values for simple and short operations.
All of the computer sciences graduated programmers, especially self-learn, no longer do assembly language or machine language, so the purpose of stack become a mystery.
The information in various installation document often does not explain how to monitor its usage, and only provide the command to set it. This post will show how to set as well as monitor it.
Check Current Stack Size
Following command can be used to check current stack size:Soft limit:
1. Login as the UNIX user you would like to check its stack size, as this setting is different for each user
2. Type "ulimit -s" or "ulimit -Ss" to display its soft limit. Following demonstrate the UNIX username is "oracle"
[oracle@centos7 ~]$ ulimit -s
10240
3. Above limit will be max size each process can reach, and will crash
4. For "oracle" user, he has ability to set it to the max limit UNIX SA allows. Type "ulimit -Hs" to show the hard limit he is allow to configure without involving SA to change the configuration file /etc/security/limits.conf
[oracle@centos7~]$ ulimit -Hs
32768
Set Maximum Stack Size
Following often find in many vendor's documentation, and every reader will be familiar with this. UNIX SA will modify /etc/security/limits.conf to set the maximum soft and hard limit for stack size
1. Login as "root" or sudo become "root"
2. Modify configuration file /etc/security/limits.conf (RedHat)
3. To set the soft and hard limit for the UNIX group "dba," to 5MB (5120) soft limit, and 10MB (10240) hard limit, perform following
# For Oracle 10.2.x - 12.2.x
@dba hard stack 10240
@dba soft stack 51204. This take effect when the user who belongs to "dba" UNIX group login. No reboot of the Linux is required. Many UNIX SA do not understand about this, and request for OS reboot and introduce additional server and application interruption
5. If there is currently running process, perform following
5.1. stop the application
5.2. logout the UNIX user, in this case, oracle
5.3. Login again as "oracle"
5.4. Check "ulimit -s" to confirm it picks up new value of 5120
5.5. Start the application, and it will limit to 5MB of stack memory
Monitor Stack Size Usage
This is the section that no vendor show how to monitor the effective stack size usage, and many vendors blindly requesting customer to increase or decrease it to try their luck. Sometimes it work, while sometimes not.Firstly, in RedHat's UNIX kernel higher stack size will indirectly causing higher virtual memory usage as it follows NPTL threading implementation.
Ref: https://access.redhat.com/solutions/227243 (public access)
I won't iterate the side effect, but let's get back to the point of monitoring a process' stack usage.
Stack size usage can be found in pseudo file /proc/
For illustration, I will use Oracle database for illustration, specially PMON daemon.
1. Check the PID of Oracle PMON daemon called "ora_pmon_"
[oracle@centos7 ~]$ ps -ef | grep pmon | grep -v grep
oracle 29012 1 0 Jan31 ? 00:02:30 ora_pmon_ORCL
2. The PID is 29012 from above output
3. Check the memory range used by stack for above PMON process/daemon in /proc//maps
[oracle@centos7 ~]$ grep stack /proc/29012/maps
7fff6d4a4000-7fff6d4b9000 rw-p 00000000 00:00 0 [stack]
4. If you are the owner of the process, then you will be able to display it. If you are not the process owner, then you will need to sudo, or run as root. Following show how to run it with sudo
[oracle@centos7 ~]$ sudo grep stack /proc/29012/maps
7fff6d4a4000-7fff6d4b9000 rw-p 00000000 00:00 0 [stack]
Similar syntax but uses awk to display the first field
awk '/stack/ {print toupper($1)}' /proc/29012/maps
7FFF6D4A4000-7FFF6D4B9000
5. Stack used memory range between 7fff6d4a4000 and 7fff6d4b9000. Subsequent steps is to find out how much memory used by using simple subtraction, but it is in hex
6. I will use build-in UNIX "bc" calculator as it is available in HP-UX, AIX, Solaris, RedHat
7. For hex value, bc needs the hex value in uppercase
[oracle@centos7 ~]$ sudo grep stack /proc/29012/maps | tr [:lower:] [:upper:]
7FFF6D4A4000-7FFF6D4B9000 RW-P 00000000 00:00 0 [STACK]
Similar syntax but uses awk to display the first firld
[oracle@centos7 ~]$ sudo awk '/stack/ {print toupper($0)}' /proc/29012/maps
7FFF6D4A4000-7FFF6D4B9000
8. Uses "ibase=16" and uppercase for the hex value to obtain the stack memory usage in bytes
[oracle@centos7 ~]$ echo "ibase=16; 7FFF6D4B9000 - 7FFF6D4A4000" | bc
86016
9. This means PMON daemon only used 86,016 bytes
10. Let's convert that to KB so that we can easily compare it with "ulimit -s" output
11. I will use awk to do the bytes to KB conversion
[oracle@centos7 ~]$ echo "ibase=16; 7FFF6D4B9000 - 7FFF6D4A4000" | bc | awk '{$a = $1/1024^1; print $a, "KB"}'
84 KB
12. Finally, let's construct a 1 line command line instead of entering multiple lines which potentially from human error
[oracle@centos7 ~]$ awk '/stack/ {print toupper($1)}' /proc/$(ps -ef | grep pmon | grep -v grep | awk '{print $2}')/maps | awk -F- '{print "ibase=16;" $2 "-" $1}'|bc | awk '{print $1/1024^1, "KB"}'
84 KB
13. Now, you can see above value is below stack soft limit of ulimit -s
$ ulimit -s
10240
14. You can see that Oracle PMON process only used 84KB, while the limit is 10240KB/process. There is no contention by PMON. If it crashed, it is not related to hitting max stack size
Verify Max Stack Size Usage Across All Users
For specific application administrator, they might be good with the information above as they knew that their application is not affected by the max stack size limit. However, as UNIX administrator, he will be concern if any of the application running in the OS is anywhere hitting the limit.
Following command will check all the currently running process and identify the top 10 processes' stack usage:
$ sudo awk '/stack/ {print toupper($1)}' /proc/*/maps | awk -F- '{print "''ibase=16; " $2 "-" $1 "''" }'|bc | sort -nr | head -10 | awk '{print $1/1024, "KB"}'
628 KB
604 KB
568 KB
552 KB
520 KB
520 KB
484 KB
476 KB
468 KB
464 KB
No comments:
Post a Comment