When you’re serving a lot of traffic it is usually the case that the traffic you’re serving is coming from a large number of local files.<\/p>\n
The kernel has built-in limits on the number of files that a process can open, and raising these limits, at a cost of some system memory, is usually a sane thing to attempt.<\/p>\n
You can view the current limit on the number of open-files by running:<\/p>\n
$ cat \/proc\/sys\/fs\/file-max\r\n<\/pre>\nThe limit can be raised interactively by running, as
root<\/code>:<\/p>\n# sysctl -w fs.file-max=100000\r\n<\/pre>\nIf you wish that change to be made persistently you should append to the file
\/etc\/sysctl.conf<\/code> the line:<\/p>\nfs.file-max = 100000\r\n<\/pre>\nThen run the following command to make your change take effect:<\/p>\n
# sysctl -p\r\n<\/pre>\nChapter 2. Socket Tuning<\/h2>\n
For servers which are handling large numbers of concurent sessions, there are some TCP options that should probabaly be tweaked.<\/p>\n
With a large number of clients comnunicating with your server it wouldn’t be unusual to have a 20,000 open sockets or more. To increase that range you append the following to the bottom of
\/etc\/sysctl.conf<\/code>:<\/p>\n# Use the full range of ports.\r\nnet.ipv4.ip_local_port_range = 1024 65535\r\n<\/pre>\nYou can also increase the recycling time of sockets, avoiding large numbers of them staying in the
TIME_WAIT<\/code> status by adding these values to\/etc\/sysctl.conf<\/code>:<\/p>\n# Enables fast recycling of TIME_WAIT sockets.\r\n# (Use with caution according to the kernel documentation!)\r\nnet.ipv4.tcp_tw_recycle = 1\r\n\r\n# Allow reuse of sockets in TIME_WAIT state for new connections\r\n# only when it is safe from the network stack\u2019s perspective.\r\nnet.ipv4.tcp_tw_reuse = 1\r\n<\/pre>\nFinally one problem you’ll find is that if a socket is listening and busy a connection-backlog will pile up. The kernel will keep pending connections in a buffer before failing. You can tweak several values to increase the size of the backlog:<\/p>\n
#\r\n# 16MB per socket - which sounds like a lot, but will virtually never\r\n# consume that much.\r\n#\r\nnet.core.rmem_max = 16777216\r\nnet.core.wmem_max = 16777216\r\n\r\n# Increase the number of outstanding syn requests allowed.\r\n# c.f. The use of syncookies.\r\nnet.ipv4.tcp_max_syn_backlog = 4096\r\nnet.ipv4.tcp_syncookies = 1\r\n\r\n# The maximum number of \"backlogged sockets\". Default is 128.\r\nnet.core.somaxconn = 1024\r\n<\/pre>\nThe trade-off here is that a connecting client will see a slow connection, but this is almost certainly better than a
Connection Refused<\/code> error.<\/p>\nOnce you’ve made those additions you can cause them to be loaded by running:<\/p>\n
# sysctl -p\r\n<\/pre>\nFinally if you’ve changed these limits you will need to restart the associated daemons. (For example «
service nginx restart<\/code>«.)<\/p>\nChapter 3. Process Scheduler<\/h2>\n
If you’re running a recent ( newer than approx 2.6.32) you’ve got the ‘Completely Fair Scheduler’ (CFS) For modern systems serving lots of connections on lots of cores, you may hit issues with process migration.<\/p>\n
There’s a kernel parameter that determines how long a migrated process has to be running before the kernel will consider migrating it again to another core. The sysctl name is
sched_migration_cost_ns<\/code>, default value 50000 (that’s ns so 0.5 ms):<\/p>\n$ cat \/proc\/sys\/kernel\/sched_migration_cost_ns\r\n<\/pre>\n(It was renamed from sched_migration_cost at some point between 3.5 and 3.8)<\/p>\n
Forking servers, like PostgreSQL or Apache, scale to much higher levels of concurrent connections if this is made larger, by at least an order of magnitude:<\/p>\n
The limit can be raised interactively by running, as
root<\/code>:<\/p>\n# sysctl -w kernel.sched_migration_cost_ns=5000000\r\n<\/pre>\nIf you wish that change to be made persistently you should append to the file
\/etc\/sysctl.conf<\/code> the line:<\/p>\nkernel.sched_migration_cost_ns = 5000000\r\n<\/pre>\nAnother parameter that can dramatically impact forking servers is
sched_autogroup_enabled<\/code>. This setting groups tasks by TTY, to improve perceived responsiveness on an interactive system. On a server with a long running forking daemon, this will tend to keep child processes from migrating away as soon as they should. It can be disabled like so:<\/p>\n# sysctl -w kernel.sched_autogroup_enabled=0\r\n<\/pre>\nVarious PostgreSQL users have reported (on the postgresql performance mailing list) gains up to 30% on highly concurrent workloads on multi-core systems.<\/p>\n
If you wish that change to be made persistently you should append to the file
\/etc\/sysctl.conf<\/code> the line:<\/p>\nkernel.sched_autogroup_enabled = 0\r\n<\/pre>\nThen run the following command to make your change take effect:<\/p>\n
# sysctl -p\r\n<\/pre>\nChapter 4. Filesystem Tuning<\/h2>\n
You almost certainly want to disable the «
atime<\/code>» option on your filesystems.<\/p>\nWith this disabled that the last time a file was accessed won’t be constantly updated every time you read a file, since this information isn’t generally useful inand causes extra disk hits, its typically disabled.<\/p>\n
To do this, just edit
\/etc\/fstab<\/code> and add «notime» as a mount option for the filesystem. For example:<\/p>\n\/dev\/rd\/c0d0p3 \/test ext3 noatime 1 2\r\n<\/pre>\nChapter 5. Swap Tuning<\/h2>\n