Basic settings to adjust for high load systems to make sure the server have enough resources to handle big number of network connections.

The main parameter is a maximum number of opened files allowed for the process to keep at the same time. Each network connection uses a file handler therefore if the limit is too low you can quickly run out of handlers and the server can not accept any more connections.

This limit is set on 2 levels - on the kernel level (fs.file-max) and on the system level (nofile).

Another kernel property which can be important in certain configurations (like transports installations or when you use proxy for Bosh connections) is: net.ipv4.ip_local_port_range. This parameter can be set the same way as the fs.file-max property.

fs.file-max = 358920
If you plan to run high load service with big number of server connections then this parameter should be at least as twice big as the number of network connections you expect to support. You can change this setting executing command:

/etc/sysctl.conf

fs.file-max=360000
net.ipv4.ip_local_port_range=1024   65000




Your IP address identifies your machine, and the port identifies a program on your machine.
A connection consists of 5 pieces of info.
    It is determined by protocol (TCP, UDP)
    local IP address and port
    remote IP address and port
for ex. a web server can service many connections on the same port.
Your webserver can even support multiple connections to the same client machine.
Say, you're connecting to google.com from two windows. Your machine will pick an unused port for each connection. So, google's server will have to keep track of (TCP, google.com, 80, yourmachine, someport1) and (TCP, google.com, 80, yourmachine, someport2).
At some point you'd bump into limits, but it's not a hard limit, and is very system dependent.

 each socket is a file descriptor, but not all machines use shorts for the fd table.
cat /proc/sys/fs/file-max gives 1606638

So, there is a limit of 65336, but it has to do with addressing, not number of connections. Number of connections is limited, but more by system config, and how much memory it has.

 64k connections per incoming IP address on a single local port

root@tc01:~# ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 20
file size               (blocks, -f) unlimited
pending signals                 (-i) 16382
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) unlimited
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
root@tc01:~#

Open files

The current value can be seen using ulimit -a (look for open files).
 set this limit by putting a file into /etc/security/limits.d/ that contains the following two lines:
*    soft    nofile    10000
*    hard    nofile    10000

Ephemeral Ports

increase the number of Ephemeral Ports available to your application.
By default this is all ports from 32768 to 61000.
We change this to all ports from 18000 to 65535.
Ports below 18000 are reserved for current and future use of the application itself.

TIME_WAIT state
TCP connections go through various states during their lifetime.
here’s the handshake that goes through multiple states, then the ESTABLISHED state, and then a whole bunch of states for either end to terminate the connection, and finally a TIME_WAIT state that lasts a really long time
netstat -tnlp

we care about is the TIME_WAIT state, and we care about it mainly because it’s so long.

By default, a connection is supposed to stay in the TIME_WAIT state for twice the msl.(maximum segment lifetime)
Its purpose is to make sure any lost packets that arrive after a connection is closed do not confuse the TCP subsystem

The default msl is 60 seconds, which puts the default TIME_WAIT timeout value at 2 minutes. Which means you’ll run out of available ports if you receive more than about 400 requests a second.

if a webserve apache/ngnix translates to 200 requests per second then it is Not good for scaling.

We fixed this by setting the timeout value to 1 second.

The two TCP tuning parameters were set using sysctl by putting a file into /etc/sysctl.d/ with the following:

net.ipv4.ip_local_port_range = 18000    65535
net.ipv4.netfilter.ip_conntrack_tcp_timeout_time_wait = 1


Connection Tracking

Note

Both the “hard” and the “soft” ulimit affect MongoDB’s performance. The “hard” ulimit refers to the maximum number of processes that a user can have active at any time. This is the ceiling: no non-root process can increase the “hard” ulimit. In contrast, the “soft” ulimit is the limit that is actually enforced for a session or process, but any process can increase it up to “hard” ulimit maximum.

A low “soft” ulimit can cause can't create new thread, closing connection errors if the number of connections grows too high. For this reason, it is extremely important to set both ulimit values to the recommended values