Monday, July 2, 2012

Too many files open

No. of Open Files
# lsof | wc -l

[root@aqua ~]# sysctl fs.file-max
fs.file-max = 10000
[root@aqua ~]# sysctl fs.file-max
fs.file-max = 10000

File Descriptor in Kernal memory
[root@aqua ~]# sysctl fs.file-nr
fs.file-nr = 2928       0       10000

a. 2828 is no. of allocated file handles
b. 0 no. of unused but alloted handles
c. 10000 system wide maximum number of file handles

No. of file handles used by a process xxx
#lsof -p xxx|wc -l
Too Many files open

use lsof command
ps aux
ps aux | grep jetty
lsof -p 1105
lsof -p 1105 | wc -l
ll /proc/11105/fd
ll /proc/1105/fd | sort -g +10 -10

To sort by the Name column
lsof -p 1105 | sort +10 -10 >/tmp/openFiles.txt

The FD column is the File Descriptor column. It is either the number of the file or one of the following:

    cwd current working directory
    Lnn library references (AIX)
    err FD information error (see NAME column)
    jld jail directory (FreeBSD)
    ltx shared library text (code and data)
    Mxx hex memory-mapped type number xx
    m86 DOS Merge mapped file
    mem memory-mapped file
    mmap memory-mapped device
    pd parent directory
    rtd root directory
    tr kernel trace file (OpenBSD)
    txt program text (code and data)
    v86 VP/ix mapped file

Determine the limit

Each user has a limit for the number of open files. This limit applies to each process run by the user. For example say the limit is 1024 and the user has three processes running, each process can open 1024 files for a total of 3072.

To determine the soft limit:
ulimit -Sn

To determine the hard limit:
ulimit -Hn

ulimit -n shows you the soft limit. The soft limit is the limit applied for opening files. The hard limit is the limit you can increase the soft limit to.

Increase the Limit

To increase the limit to 10000 use the following command:

ulimit -Sn 10000

You can change the hard limit too, ulimit -Hn 10000. ulimit -n 10000 changes both the soft and hard limits to the same value.

Note: Once you reboot the limit is reset.
You cannot determine the limit of the root user using ulimit. For example:

sudo ulimit -n
sudo: ulimit: command not found
To make the limits bigger and to make the change permanent, edit your configuration file and reboot.

You can use * in the limit.conf file instead of a user name to specify all users, however this does not apply to the root!

    soft nofile 10000
    hard nofile 10000
    The limit.conf file is applied during the boot process. 
    You cannot start a process late enough in the boot process! For example: “sudo update-rc.d tomcat defaults 99 01″ is at the end and it is still too late.

The work around is to force the limit to be set before starting the process. Put “ulimit -n 4000″ before starting your process, then the limit.conf file is processed here.

There is another file limit in the system, the total number of files that can be opened by all processes.

To see the file max value:
sysctl -a | grep fs.file-max
fs.file-max = 170469 
File descriptor can be limited in both system level and shell level.
To check maximum number of fd in system type 'cat /proc/sys/fs/file-max'.
If u want to set it add 'fs.file-max = 200000' to /etc/sysctl.conf
To check shell level limit
Shell level 'ulimit -n' 


In /etc/security/limits.conf
* soft nofile 2048
* hard nofile 2048
To get total number of handle that a process use
lsof -p PID |wc -l
For File Descriptors == >  Easy way to edit the file
echo "#### Custom Addedlines ####">> /etc/security/limits.conf 
echo "* soft nofile 32768" | sudo tee -a /etc/security/limits.conf
echo "* hard nofile 32768" | sudo tee -a /etc/security/limits.conf
echo "root soft nofile 32768" | sudo tee -a /etc/security/limits.conf
echo "root hard nofile 32768" | sudo tee -a /etc/security/limits.conf
For User limits  == > 

* soft nofile 32768
* hard nofile 32768
root soft nofile 32768
root hard nofile 32768
* soft memlock unlimited
* hard memlock unlimited
root soft memlock unlimited
root hard memlock unlimited
* soft as unlimited
* hard as unlimited
root soft as unlimited
root hard as unlimited
In addition, you may need to be run the following command:
sysctl -w vm.max_map_count=131072
The command enables more mapping. It is not in the limits.conf file.

Disable Swap

Disable swap entirely. This prevents the Java Virtual Machine (JVM) from responding poorly because it is buried in swap and ensures that the OS OutOfMemory (OOM) killer does not kill

sudo swapoff --all

echo "*   soft    nofile  131070" >> /etc/security/limits.conf
echo "*   hard    nofile  131070" >> /etc/security/limits.conf

echo 0 > /proc/sys/vm/swappiness
echo "vm.swappiness=0" >> /etc/sysctl.conf

Both the “hard” and the “soft” ulimit affect MongoDB’s performance. The “hard” ulimit refers to the maximum number of processes that a user can have active at any time. This is the ceiling: no non-root process can increase the “hard” ulimit. In contrast, the “soft” ulimit is the limit that is actually enforced for a session or process, but any process can increase it up to “hard” ulimit maximum.
A low “soft” ulimit can cause can't create new thread, closing connection errors if the number of connections grows too high. For this reason, it is extremely important to set both ulimit values to the recommended values
sudo vim /etc/security/limits.conf

*           soft    nofile         10000
*           hard    nofile         20000
Modify the common-session file with the following: sudo vim /etc/pam.d/common-session
session required
you can set system wide limits by editing /etc/security/limits.conf and /etc/pam.d/common-session, however, this only applies interactive and non-interactive shells (and processes started by them).
If you want to try system wide limits, then add a line like the following to /etc/security/limits.conf:
*  -  nofile 65000

In /etc/pam.d/common-session, enable the limits module by adding:
session required

Keep in mind that all this really does is have PAM set the limit for 
interactive and non-interactive shells when loaded. Processes then 
pickup these limits when started from shells/scripts. You should 
probably do a restart to fully enact these settings. 

and reboot the system.

The option with wildcard *didn’t work for me , because I used root accout to run my programms and wildcard option doesn’t affect the superuser.


No comments:

Post a Comment