Too many files open

No. of Open Files
# lsof | wc -l

[root@aqua ~]# sysctl fs.file-max
fs.file-max = 10000
[root@aqua ~]# sysctl fs.file-max
fs.file-max = 10000

File Descriptor in Kernal memory
[root@aqua ~]# sysctl fs.file-nr
fs.file-nr = 2928       0       10000

a. 2828 is no. of allocated file handles
b. 0 no. of unused but alloted handles
c. 10000 system wide maximum number of file handles

No. of file handles used by a process xxx

#lsof -p xxx|wc -l

=================

Too Many files open

use lsof command
ps aux
ps aux | grep jetty
lsof -p 1105
lsof -p 1105 | wc -l
ll /proc/11105/fd
ll /proc/1105/fd | sort -g +10 -10

To sort by the Name column
lsof -p 1105 | sort +10 -10 >/tmp/openFiles.txt

The FD column is the File Descriptor column. It is either the number of the file or one of the following:

    cwd current working directory
    Lnn library references (AIX)
    err FD information error (see NAME column)
    jld jail directory (FreeBSD)
    ltx shared library text (code and data)
    Mxx hex memory-mapped type number xx
    m86 DOS Merge mapped file
    mem memory-mapped file
    mmap memory-mapped device
    pd parent directory
    rtd root directory
    tr kernel trace file (OpenBSD)
    txt program text (code and data)
    v86 VP/ix mapped file

Determine the limit

Each user has a limit for the number of open files. This limit applies to each process run by the user. For example say the limit is 1024 and the user has three processes running, each process can open 1024 files for a total of 3072.

To determine the soft limit:
ulimit -Sn
1024

To determine the hard limit:
ulimit -Hn
2048

ulimit -n shows you the soft limit. The soft limit is the limit applied for opening files. The hard limit is the limit you can increase the soft limit to.

Increase the Limit

To increase the limit to 10000 use the following command:

ulimit -Sn 10000

You can change the hard limit too, ulimit -Hn 10000. ulimit -n 10000 changes both the soft and hard limits to the same value.

Note: Once you reboot the limit is reset.
You cannot determine the limit of the root user using ulimit. For example:

sudo ulimit -n
sudo: ulimit: command not found
To make the limits bigger and to make the change permanent, edit your configuration file and reboot.

You can use * in the limit.conf file instead of a user name to specify all users, however this does not apply to the root!

    soft nofile 10000
    hard nofile 10000
    
    The limit.conf file is applied during the boot process. 
    
    
    You cannot start a process late enough in the boot process! For example: “sudo update-rc.d tomcat defaults 99 01″ is at the end and it is still too late.

The work around is to force the limit to be set before starting the process. Put “ulimit -n 4000″ before starting your process, then the limit.conf file is processed here.

There is another file limit in the system, the total number of files that can be opened by all processes.

To see the file max value:
sysctl -a | grep fs.file-max
fs.file-max = 170469 

File descriptor can be limited in both system level and shell level.

To check maximum number of fd in system type 'cat /proc/sys/fs/file-max'.

730437

If u want to set it add 'fs.file-max = 200000' to /etc/sysctl.conf

To check shell level limit

Shell level 'ulimit -n' 

1024

In /etc/security/limits.conf

* soft nofile 2048
* hard nofile 2048

To get total number of handle that a process use

lsof -p PID |wc -l

For File Descriptors == >  Easy way to edit the file

echo "#### Custom Addedlines ####">> /etc/security/limits.conf 

echo "* soft nofile 32768" | sudo tee -a /etc/security/limits.conf
echo "* hard nofile 32768" | sudo tee -a /etc/security/limits.conf
echo "root soft nofile 32768" | sudo tee -a /etc/security/limits.conf
echo "root hard nofile 32768" | sudo tee -a /etc/security/limits.conf

For User limits  == > 

* soft nofile 32768
* hard nofile 32768
root soft nofile 32768
root hard nofile 32768
* soft memlock unlimited
* hard memlock unlimited
root soft memlock unlimited
root hard memlock unlimited
* soft as unlimited
* hard as unlimited
root soft as unlimited
root hard as unlimited

In addition, you may need to be run the following command:

sysctl -w vm.max_map_count=131072

The command enables more mapping. It is not in the limits.conf file.

Disable Swap

Disable swap entirely. This prevents the Java Virtual Machine (JVM) from responding poorly because it is buried in swap and ensures that the OS OutOfMemory (OOM) killer does not kill

sudo swapoff --all

OR

echo "*   soft    nofile  131070" >> /etc/security/limits.conf
echo "*   hard    nofile  131070" >> /etc/security/limits.conf

echo 0 > /proc/sys/vm/swappiness
echo "vm.swappiness=0" >> /etc/sysctl.conf

Note
Both the “hard” and the “soft” ulimit affect MongoDB’s
performance. The “hard” ulimit refers to the maximum number of
processes that a user can have active at any time. This is the
ceiling: no non-root process can increase the “hard” ulimit. In
contrast, the “soft” ulimit is the limit that is actually
enforced for a session or process, but any process can increase it
up to “hard” ulimit maximum.



A low “soft” ulimit can cause can't create new thread,
closing connection errors if the number of connections
grows too high. For this reason, it is extremely important to set
both ulimit values to the recommended values
 

 OR

sudo vim /etc/security/limits.conf

*           soft    nofile         10000

*           hard    nofile         20000







Modify the common-session file with the following:
sudo vim /etc/pam.d/common-session




session required pam_limits.so




==



you can set system wide limits by editing /etc/security/limits.conf and 
/etc/pam.d/common-session, however, this only applies interactive and 
non-interactive shells (and processes started by them).



If you want to try system wide limits, then add a line like the following to /etc/security/limits.conf:



*  -  nofile 65000



In /etc/pam.d/common-session, enable the limits module by adding:



session required pam_limits.so



Keep in mind that all this really does is have PAM set the limit for 
interactive and non-interactive shells when loaded. Processes then 
pickup these limits when started from shells/scripts. You should 
probably do a restart to fully enact these settings. 




and reboot the system.


The option with wildcard *didn’t work for me , because I used root accout to run my programms and wildcard option doesn’t affect the superuser.