UNIX ulimit Settings

Most UNIX-like operating systems, including Linux and macOS, provideways to limit and control the usage of system resources such asthreads, files, and network connections on a per-process and per-userbasis. These “ulimits” prevent single users from using too many systemresources. Sometimes, these limits have low default values that cancause a number of issues in the course of normal MongoDB operation.

Resource Utilization

mongod and mongos each use threads and filedescriptors to track connections and manage internal operations. Thissection outlines the general resource utilization patterns for MongoDB.Use these figures in combination with the actual information about yourdeployment and its use to determine ideal ulimit settings.

Generally, all mongod and mongos instances:

• track each incoming connection with a file descriptor and athread.
• track each internal thread or pthread as a system process.

mongod

• 1 file descriptor for each data file in use by themongod instance.
• 1 file descriptor for each journal file used by themongod instance when storage.journal.enabled is true.
• In replica sets, each mongod maintains a connection toall other members of the set.

mongod uses background threads for a number of internalprocesses, including TTL collections,replication, and replica set health checks, which may require a smallnumber of additional resources.

mongos

In addition to the threads and file descriptors for client connections,mongos must maintain connections to all config servers andall shards, which includes all members of all replica sets.

For mongos, consider the following behaviors:

• mongos instances maintain a connection pool to each shardso that the mongos can reuse connections and quicklyfulfill requests without needing to create new connections.

• You can limit the number of incoming connections usingthe net.maxIncomingConnections run-time option.By restricting the number of incoming connections you can prevent acascade effect where the mongos creates too manyconnections on the mongod instances.

Note

Changed in version 2.6: MongoDB removed the upward limit on the maxIncomingConnectionssetting.

Review and Set Resource Limits

ulimit

You can use the ulimit command at the system prompt to checksystem limits, as in the following example:

$ ulimit -a
-t: cpu time (seconds)         unlimited
-f: file size (blocks)         unlimited
-d: data seg size (kbytes)     unlimited
-s: stack size (kbytes)        8192
-c: core file size (blocks)    0
-m: resident set size (kbytes) unlimited
-u: processes                  192276
-n: file descriptors           21000
-l: locked-in-memory size (kb) unlimited
-v: address space (kb)         unlimited
-x: file locks                 unlimited
-i: pending signals            192276
-q: bytes in POSIX msg queues  819200
-e: max nice                   30
-r: max rt priority            65
-N 15:                         unlimited

ulimit refers to the per-user limitations for variousresources. Therefore, if your mongod instance executes as auser that is also running multiple processes, or multiplemongod processes, you might see contention for theseresources. Also, be aware that the processes value (i.e. -u)refers to the combined number of distinct processes and sub-processthreads.

You can change ulimit settings by issuing a command in thefollowing form:

ulimit -n <value>

There are both “hard” and the “soft” ulimits that affect MongoDB’sperformance. The “hard” ulimit refers to the maximum number ofprocesses that a user can have active at any time. This is theceiling: no non-root process can increase the “hard” ulimit. Incontrast, the “soft” ulimit is the limit that is actuallyenforced for a session or process, but any process can increase itup to “hard” ulimit maximum.

A low “soft” ulimit can cause can't create new thread,closing connection errors if the number of connectionsgrows too high. For this reason, it is extremely important to setboth ulimit values to the recommended values.

ulimit will modify both “hard” and “soft” values unless the-H or -S modifiers are specified whenmodifying limit values.

For many distributions of Linux you can change values by substitutingthe -n option for any possible value in the output of ulimit-a. On macOS, use the launchctl limit command. See youroperating system documentation for the precise procedure for changingsystem limits on running systems.

After changing the ulimit settings, you must restart theprocess to take advantage of the modified settings. You can use the/proc file system to see the current limitations on a runningprocess.

Depending on your system’s configuration, and default settings, anychange to system limits made using ulimit may revert followinga system restart. Check your distribution and operatingsystem documentation for more information.

SUSE Linux Enterprise Server

SUSE Linux Enterprise Server and potentially other SUSE distributions shipwith virtual memory address space limited to 8 GB by default. You _must_adjust this in order to prevent virtual memory allocation failures as thedatabase grows.

The SLES packages for MongoDB automatically adjust these limits intheir default init script. If you are starting MongoDB manually withoutthe provided init script, are using your own custom init script, orare using the TGZ tarball release, you must make these changesyourself.

Red Hat Linux Enterprise Server and CentOS

Red Hat Enterprise Linux and CentOS 6 and 7 enforce a separate max processlimitation, nproc, which overrides ulimit settings. This value isdefined in the following configuration file, depending on version:

To configure an nproc value for these versions, create a file named/etc/security/limits.d/99-mongodb-nproc.conf with new soft nproc andhard nproc values to increase the process limit. For recommended values,see Recommended ulimit Settings.

With RHEL / CentOS 8, separate nproc values are no longer necessary.The ulimit command is sufficient to configure the required maxprocess values on RHEL / CentOS 8.

Recommended ulimit Settings

Every deployment may have unique requirements and settings; however,the following thresholds and settings are particularly important formongod and mongos deployments:

• -f (file size): unlimited
• -t (cpu time): unlimited
• -v (virtual memory): unlimited [1]
• -l (locked-in-memory size): unlimited
• -n (open files): 64000
• -m (memory size): unlimited [1][2]
• -u (processes/threads): 64000

Always remember to restart your mongod andmongos instances after changing the ulimit settings toensure that the changes take effect.

Linux distributions using Upstart

For Linux distributions that use Upstart, you can specify limitswithin service scripts if you start mongod and/ormongos instances as Upstart services. You can do this byusing limit stanzas.

Specify the Recommended ulimit Settings, as in the followingexample:

limit fsize unlimited unlimited    # (file size)
limit cpu unlimited unlimited      # (cpu time)
limit as unlimited unlimited       # (virtual memory size)
limit memlock unlimited unlimited  # (locked-in-memory size)
limit nofile 64000 64000           # (open files)
limit nproc 64000 64000            # (processes/threads)

Each limit stanza sets the “soft” limit to the first value specified and the “hard”limit to the second.

After changing limit stanzas, ensure that the changes takeeffect by restarting the application services, using the followingform:

restart <service name>

Linux distributions using systemd

For Linux distributions that use systemd, you can specifylimits within the [Service] sections of service scripts if youstart mongod and/ormongos instances as systemd services. You can do thisby using resource limit directives.

Specify the Recommended ulimit Settings, as in the followingexample:

[Service]
# Other directives omitted
# (file size)
LimitFSIZE=infinity
# (cpu time)
LimitCPU=infinity
# (virtual memory size)
LimitAS=infinity
# (locked-in-memory size)
LimitMEMLOCK=infinity
# (open files)
LimitNOFILE=64000
# (processes/threads)
LimitNPROC=64000

Each systemd limit directive sets both the “hard” and “soft” limits to the valuespecified.

After changing limit stanzas, ensure that the changes takeeffect by restarting the application services, using the followingform:

systemctl restart <service name>

/proc File System

Note

This section applies only to Linux operating systems.

The /proc file-system stores the per-process limits in thefile system object located at /proc/<pid>/limits, where <pid>is the process’s PID or process identifier. You can use thefollowing bash function to return the content of the limitsobject for a process or processes with a given name:

return-limits(){
 
     for process in $@; do
          process_pids=`ps -C $process -o pid --no-headers | cut -d " " -f 2`
 
          if [ -z $@ ]; then
             echo "[no $process running]"
          else
             for pid in $process_pids; do
                   echo "[$process #$pid -- limits]"
                   cat /proc/$pid/limits
             done
          fi
 
     done
 
}

You can copy and paste this function into a current shell session orload it as part of a script. Call the function with one the followinginvocations:

return-limits mongod
return-limits mongos
return-limits mongod mongos