I get errors about ulimit in splunkd.log

Problem

Are you seeing messages about ulimits in splunkd.log while running Splunk software on *nix? Are those hosts experiencing a Splunk software crash?

03-03-2011 21:50:09.027 INFO  ulimit - Limit: virtual address space size: unlimited
03-03-2011 21:50:09.027 INFO  ulimit - Limit: data segment size: 1879048192 bytes [hard maximum: unlimited]
03-03-2011 21:50:09.027 INFO  ulimit - Limit: resident memory size: 2147482624 bytes [hard maximum: unlimited]
03-03-2011 21:50:09.027 INFO  ulimit - Limit: stack size: 33554432 bytes [hard maximum: 2147483646 bytes]
03-03-2011 21:50:09.027 INFO  ulimit - Limit: core file size: 1073741312 bytes [hard maximum: unlimited]
03-03-2011 21:50:09.027 INFO  ulimit - Limit: data file size: 2147483646 bytes
03-03-2011 21:50:09.027 ERROR ulimit - Splunk may not work due to low file size limit
03-03-2011 21:50:09.027 INFO  ulimit - Limit: open files: 1024
03-03-2011 21:50:09.027 INFO  ulimit - Limit: cpu time: unlimited
03-03-2011 21:50:09.029 INFO  loader - Splunkd starting (build 95063).

If so, you might need to adjust your server ulimit settings. The ulimits control the resources available to a *nix shell and any processes that shell starts. A *nix host running Splunk software often needs a higher ulimit setting than the defaults.

Causes

First, determine the ulimit settings on the host that has an issue. There are a few ways you can check your ulimit settings.

Each time the Splunk Enterprise service is started or restarted, it will report on the ulimits. You can search the internal logs for the report using:
index=_internal source=*splunkd.log ulimit
.
On the *nix command line, find the main Splunk Enterprise process (splunkd) and check the ulimits for that process by running prlimit --pid=$pid
The monitoring console includes a health check for ulimits. See Access and customize health check in Monitoring Splunk Enterprise.

Next, compare the ulimits reported to the guidance available for Splunk Enterprise.

For the minimum ulimit setting recommendations, see Considerations regarding system-wide resource limits on *nix systems in the Installation Manual.
Determine if your organization has internal guidance on ulimits settings for *nix hosts. And if there is, what ulimit values are set and how the settings maintained and distributed across the *nix hosts.

Finally, plan any necessary updates to the ulimit settings using the preferred change management option, and document the changes in a change control.

Solutions

You can set new ulimit values by updating the *nix host configuration files.

The methods of setting ulimit values will vary based on the *nix distribution and the distribution's service management tool. Review the distribution documentation before updating the ulimits.
You will need to work with your system administrator to set persistent ulimit values.

Set limits using /etc/security/limits.conf

These instructions are for *nix distributions that utilize the init service.

Become the root user or an administrative equivalent with su:
```
sudo su -
```
Open /etc/security/limits.conf with a text editor.

Add or change the values in the file. Example:

*  hard  nofile   64000
*  hard  nproc   16000
*  hard  fsize     -1

Save the file and exit the text editor.
Restart the machine to complete the changes.

Set limits using the /etc/systemd configuration files

The *nix distributions that utilize the systemd service can have ulimits set at the system level, user level, and service level.

The /etc/systemd/system.conf file sets system-wide limits.
The /etc/systemd/user.conf file sets limits for services that run under a specific user.
The /etc/systemd/system/Splunkd.service file is a service-level configuration that can be created during the Splunk Software installation process or manually. See Configure systemd using enable boot-start in the Admin Manual.

Become the root user or an administrative equivalent with su:
```
sudo su -
```
Open the configuration file with a text editor. Example: /etc/systemd/system/Splunkd.service