Support for custom search commands

How to use custom search commands in Splunk Federated Search for Splunk deployments in standard mode.

When using custom search commands in Splunk Federated Search for Splunk deployments in standard mode, you might need to be aware of the location where the custom search commands run. For most custom commands, search results aren't affected by where they are run. However, there are some cases where you might get incorrect results if a custom command runs on the wrong search head.

See custom search commands documentation.

Run custom commands on federated or remote search heads

Each custom command can be configured with an argument called local in the commands.conf file. By default, this argument is set to false, which means the custom command can be sent to and run on the remote search head in a federated search. If the local argument for a custom command is set to true, the command will run on the federated search head instead. That said, there is a possibility that the command could still run on the remote search head even if local has a value of true.

Although the local setting provides flexibility, you must use the local setting carefully. For example, local must be set to false for certain custom commands that must run on the remote search head because they meet the following conditions:
  • The command requires preprocessing, which means it must run in the streaming phase as defined by the requires_preop argument in the commands.conf file.
  • The command must be the first command listed in phase 1 of the generated phases of a search. To identify which phase a command is running in, check the Search job properties in the Search Job Inspector. See View search job properties.

In most cases, even when a custom command requires the requires_preop argument and is set to local=true, the command might still be able to run successfully and the search might complete as expected. However, the exception arises when the custom command must be the first command in phase 1 of the search and requires preprocessing. Your searches might fail or produce incorrect results if a custom command that requires preop and must run as the first phase in the search is forced by the local=true setting to run on the federated search head. This is because the necessary preprocessing for these custom commands can’t happen on the local federated search head and must occur on the remote search head instead.

In these situations, Splunk platform sends the entire search to the remote search head for processing and displays a warning message informing the user that the command can’t be run on the federated search head. To resolve this issue, users should either modify the search to place a non-streaming command before the custom command or use a different command that does not require streaming preprocessing. Understanding these behaviors ensures that federated searches return accurate results and helps users avoid configuration conflicts.

Turn on the local argument

Splunk Cloud Platform

To turn on the local argument in your environment, request help from Splunk Support. If you have a support contract, file a new case using the Splunk Support Portal at Support and Services. Otherwise, contact Splunk Customer Support.

Splunk Enterprise

To turn the local argument on in your environment, follow these steps.

Prerequisites
  • Have the permissions to edit configuration files. Only users with file system access, such as system administrators, can edit configuration files.
  • Know how to edit configuration files. Review the steps in How to edit a configuration file in the Splunk Enterprise Admin Manual.
  • Decide which directory to store configuration file changes in. There can be configuration files with the same name in your default, local, and app directories. See Where you can place (or find) your modified configuration files in the Splunk Enterprise Admin Manual.
CAUTION: Never change or copy the configuration files in the default directory. The files in the default directory must remain intact and in their original location. Make changes to the files in the local directory.

Steps

  1. Open or create a local commands.conf file at $SPLUNK_HOME/etc/system/local. If your custom search script is in the $SPLUNK_HOME/etc/apps/MY_APP/bin/path, put a custom commands.conf file in the $SPLUNK_HOME/etc/apps/MY_APP/default/ directory.
  2. In the stanza for the custom command, add the line local=true.

Troubleshoot custom commands on remote search heads

In certain situations, if a custom command is not defined on the remote provider, the remote search head will fail the search during the parsing phase. As a result, if users attempt to run a custom search command on the remote search head and the command definition does not exist there, the search might fail. If you encounter a failure when running a search that uses a custom command, an important troubleshooting step is to check whether the custom command is properly defined on the remote search head.