Linux ulimit: Setting and Monitoring Stack Size

Linux ulimit: All about Process Stack Size

In many application installation, e.g. SAP BusinessObject, Oracle RDBMS, IBM Cognos Analytics, there is always a section about ulimit configuration like below:

$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 127458
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 4096
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 2047
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

In this post, I will focus in "stack size" configuration, and monitoring.

Stack is the maximum allowable size, in KB, that this specific UNIX user allow for each of the UNIX process.  If it is multi-threaded, then all threads going to share the same stack size,  In other word, the stack size will be total of all threads consumed.  Default, RedHat/CentOS/Fedora set it to 10 MB.

For stack usage, 10MB is consider very high, as it is a memory area that used to perform stack push and pop (research assembly language PUSH and POP for more detail) values for simple and short operations.

All of the computer sciences graduated programmers, especially self-learn, no longer do assembly language or machine language, so the purpose of stack become a mystery.

The information in various installation document often does not explain how to monitor its usage, and only provide the command to set it.  This post will show how to set as well as monitor it.

Check Current Stack Size

Following command can be used to check current stack size:

Soft limit:
1. Login as the UNIX user you would like to check its stack size, as this setting is different for each user
2. Type "ulimit -s" or "ulimit -Ss" to display its soft limit.  Following demonstrate the UNIX username is "oracle"

[oracle@centos7 ~]$ ulimit -s
10240

3. Above limit will be max size each process can reach, and will crash
4. For "oracle" user, he has ability to set it to the max limit UNIX SA allows. Type "ulimit -Hs" to show the hard limit he is allow to configure without involving SA to change the configuration file /etc/security/limits.conf

[oracle@centos7~]$ ulimit -Hs
32768

Set Maximum Stack Size

Following often find in many vendor's documentation, and every reader will be familiar with this.  UNIX SA will modify /etc/security/limits.conf to set the maximum soft and hard limit for stack size

1. Login as "root" or sudo become "root"

2. Modify configuration file /etc/security/limits.conf (RedHat)

3. To set the soft and hard limit for the UNIX group "dba," to 5MB (5120) soft limit, and 10MB (10240) hard limit, perform following

# For Oracle 10.2.x - 12.2.x
@dba          hard    stack              10240
@dba          soft    stack              5120

4. This take effect when the user who belongs to "dba" UNIX group login.  No reboot of the Linux is required.  Many UNIX SA do not understand about this, and request for OS reboot and introduce additional server and application interruption
5. If there is currently running process, perform following
5.1. stop the application
5.2. logout the UNIX user, in this case, oracle
5.3. Login again as "oracle"
5.4. Check "ulimit -s" to confirm it picks up new value of 5120
5.5. Start the application, and it will limit to 5MB of stack memory

Monitor Stack Size Usage

This is the section that no vendor show how to monitor the effective stack size usage, and many vendors blindly requesting customer to increase or decrease it to try their luck.  Sometimes it work, while sometimes not.

Firstly, in RedHat's UNIX kernel higher stack size will indirectly causing higher virtual memory usage as it follows NPTL threading implementation.

Ref: https://access.redhat.com/solutions/227243 (public access)

I won't iterate the side effect, but let's get back to the point of monitoring a process' stack usage.

Stack size usage can be found in pseudo file /proc//maps under a line called "stack."  There will be only 1 line in this file.

For illustration, I will use Oracle database for illustration, specially PMON daemon.

1. Check the PID of Oracle PMON daemon called "ora_pmon_"

[oracle@centos7 ~]$ ps -ef | grep pmon | grep -v grep
oracle   29012     1  0 Jan31 ?        00:02:30 ora_pmon_ORCL

2. The PID is 29012 from above output

3. Check the memory range used by stack for above PMON process/daemon in /proc//maps

[oracle@centos7 ~]$ grep stack /proc/29012/maps

7fff6d4a4000-7fff6d4b9000 rw-p 00000000 00:00 0   [stack]

4. If you are the owner of the process, then you will be able to display it.  If you are not the process owner, then you will need to sudo, or run as root.  Following show how to run it with sudo

[oracle@centos7 ~]$ sudo grep stack /proc/29012/maps

7fff6d4a4000-7fff6d4b9000 rw-p 00000000 00:00 0   [stack]

Similar syntax but uses awk to display the first field

awk '/stack/ {print  toupper($1)}' /proc/29012/maps

7FFF6D4A4000-7FFF6D4B9000

5. Stack used memory range between 7fff6d4a4000 and 7fff6d4b9000.  Subsequent steps is to find out how much memory used by using simple subtraction, but it is in hex

6. I will use build-in UNIX "bc" calculator as it is available in HP-UX, AIX, Solaris, RedHat

7. For hex value, bc needs the hex value in uppercase

[oracle@centos7 ~]$ sudo grep stack /proc/29012/maps  | tr [:lower:] [:upper:]

7FFF6D4A4000-7FFF6D4B9000 RW-P 00000000 00:00 0    [STACK]

Similar syntax but uses awk to display the first firld

[oracle@centos7 ~]$ sudo awk '/stack/ {print  toupper($0)}' /proc/29012/maps

7FFF6D4A4000-7FFF6D4B9000

8. Uses "ibase=16" and uppercase for the hex value to obtain the stack memory usage in bytes

[oracle@centos7 ~]$ echo "ibase=16; 7FFF6D4B9000 - 7FFF6D4A4000" | bc

86016

9. This means PMON daemon only used 86,016 bytes

10. Let's convert that to KB so that we can easily compare it with "ulimit -s" output

11. I will use awk to do the bytes to KB conversion

[oracle@centos7 ~]$ echo "ibase=16; 7FFF6D4B9000 - 7FFF6D4A4000" | bc | awk '{$a = $1/1024^1; print $a, "KB"}'

84 KB

12. Finally, let's construct a 1 line command line instead of entering multiple lines which potentially from human error

[oracle@centos7 ~]$ awk '/stack/ {print  toupper($1)}' /proc/$(ps -ef | grep pmon | grep -v grep | awk '{print $2}')/maps | awk -F- '{print "ibase=16;" $2 "-"  $1}'|bc | awk '{print $1/1024^1, "KB"}'

84 KB

13. Now, you can see above value is below stack soft limit of ulimit -s

$ ulimit -s

10240

14. You can see that Oracle PMON process only used 84KB, while the limit is 10240KB/process.  There is no contention by PMON.  If it crashed, it is not related to hitting max stack size

Verify Max Stack Size Usage Across All Users

For specific application administrator, they might be good with the information above as they knew that their application is not affected by the max stack size limit.  However, as UNIX administrator, he will be concern if any of the application running in the OS is anywhere hitting the limit.

Following command will check all the currently running process and identify the top 10 processes' stack usage:

$ sudo awk '/stack/ {print  toupper($1)}' /proc/*/maps | awk -F- '{print "''ibase=16; " $2 "-"  $1 "''" }'|bc | sort -nr | head -10 | awk '{print $1/1024, "KB"}'

628 KB

604 KB

568 KB

552 KB

520 KB

520 KB

484 KB

476 KB

468 KB

464 KB

UNIX: Disabling SSH Client Session Disconnect on Idle

UNIX: Disabling/Extending SSH Client Session Disconnect on Idle

Overview

Typical organization will disconnect idle SSH connection when idle.  Hours of searching in Internet, and found a lots of partial answer in extending ssh client session getting disconnected.  Therefore, I written this post to summarize all, and mainly focus in RedHat RHEL 7.x (CentOS 7.x) as this is the environment that typically involve for ssh (Windows ssh support only started with late Windows 10 and Windows 2019 Server so I'm not going to include).

Following are various approach a typical organization will configure to kick out idle ssh connections:

1. Firewall is terminating idle ssh connection - firewall could be in the UNIX server, or sit between UNIX server and ssh client
2. bash shell is terminating idle sh connection - Linux bash shell has environment parameter TMOUT which can be configured to disconnect idle sh.  To ssh user, they will see their ssh getting disconnected, but /var/log/secure (sshd daemon log file) will not show an entry on disconnection
3. sshd daemon in UNIX server terminating idle ssh connection - There are 2 parameters that allows sshd server to disconnect idle ssh client connection

Some information found in Internet often covered one of the 3 above, while users might experiencing all 3.  Inexperience administrators often confused how to troubleshoot in order to fix it, while some over-configured and resulting more administration overhead

Configuration Overview

Following are the necessary configuration, and log files in troubleshooting each of the above 3 setup:

#	Component	Configuration File	Parameter	Value	Comment
1	Firewall/router	Vendor dependent	Vendor dependent	Vendor dependent	firewalld & iptables used by RHEL do not have capability to drop idle connection. So this configuration is in external firewall device, such as Cisco firewall/router
2	bash	/etc/profile ~/.bash_profile ~/.profile	TMOUT	None (default), or set to desire second	Type "unset TMOUT" to remove this parameter, and bash won't terminate idle session Following message will display in console if it reached TMOUT: timed out waiting for input: auto-logout
3	sshd (Server)	/etc/ssh/sshd_config	1. ClientAliveInterval 2. ClientAliveCountMax	Comment both out, and restart sshd (service restart sshd). To enable it, set ClientAliveCountMax=0, and specify the desired timeout value as ClientAliveInterval. E.g. to set 15 min timeout, set ClientAliveInterval=15m (yes, m is acceptable as # of minute)	Following message will appear in /var/log/secure when sshd closing idle connection Jan 29 11:48:15 myhostname sshd[9614]: Timeout, client not responding.

UNIX administrator might enabled firewall & router to kill idle connection.  If it is not possible to convince that team to adjust it to reasonable time, then you can configure ssh client to send keep-alive message to server to fool the firewall & router and won't get kick off

SSH client parameter:

1. ServerAliveInterval = 5m

Above will send a TCP NULL packet to sshd daemon every 5 min, and both firewall and router will think someone is actively entering something.  The connection is treated as active, and won't be terminated

Wrong Configuration

Scenario 1: /etc/ssh/sshd_config (server) and ~/.ssh/ssh_config (client) have configured tcpkeepalive=yes, but session still get disconnected

Explanation: Firstly, find out what is kicking you out.  If it is sshd, then /var/log/secure will have an entry as shown above.  If it is bash, then type "echo $TMOUT" to see whether it is configured.  If both are not configured, then it is very likely the firewall/router.

Next, find out how soon the ssh client session is kick out.  Open a new ssh session to server, and check back every 5 min.  You should be able to get a brief idea, so that you have a baseline to test with.

If the session getting kick out after 5 min, then configure the ssh client to enable keepalive.  PuTTY has this in the option menu, and ssh client too.

Use PuTTY (ssh client) to open a new connection to server by enabling keepalive, and wait for 5 min.  If it is not getting disconnected, then you have a temporary workaround, while investigating the root cause.

For Linux client, configure /etc/ssh/ssh_config, parameter ServerAliveInterval=x, where x is # of seconds to send TCP NULL packet to sshd server

Scenario 2: /etc/ssh/sshd_config (server) has ClientAliveInterval=600m, yet ssl client keep getting disconnected.  There is no firewall/router/bash timeout.  What could be the problem

Multiple value of ClientAliveInterval in /etc/ssh/sshd_config could be the cause.  Some people blindly add ClientAliveInterval=600m to the end of the config file, while sshd read the first entry on top of the configuration file.  Therefore, the effective setting is still the old value (on top).

Second reason could be sshd daemon is not restarted after configuration change.  sshd doesn't activate the new setting until it get restarted using "service restart sshd"

Type "sshd -T | grep clientalive" to see the effective setting

Scenario 3: /etc/ssh/sshd_config (server) has been configured to comment out both ClientAlive* setting, which won't disconnect idle ssh client session.  Why my ssh session still get disconnected

In /etc/profile, environment variable "TMOUT" could be configured, which will kick idle session out as well.  Once open a new ssh session, type "echo $TMOUT" to check whether it is configured.  Remove the timeout using following:

$ unset TMOUT

You can reset it in ~/.bash_profile so that it will override the global default in /etc/profile.  Fill in this line in .bash_profile

unset TMOUT