Run standard mode federated searches over lookups
For federated searches over standard mode that use the lookup command, Splunk software optimizes processing of the lookup on the federated search head of your local Splunk platform deployment and the remote search head of federated providers depending on the specific conditions of the search. For search performance reasons, Splunk software processes searches with the lookup command on the remote search heads of the federated providers when possible. Most simple searches with lookups are run only on remote search heads.
If you are running federated searches over standard mode Splunk platform federated providers, and you want to use the lookup command to enrich the results of a federated search, the following conditions must be met for the search to return results.
- The lookup definition and lookup table expected by the
lookupcommand must exist on the remote search heads. For some types of complex searches with thelookupcommand, the lookup definition and lookup table must also exist on the federated search heads. - The service account on remote search heads of your federated providers must have access permissions for the lookup definition and lookup table that are on the remote search heads. See Service accounts and security for Federated Search for Splunk.
For example, before a search that uses the lookup command and calculates aggregate statistics is sent to the remote search head, the local federated search head first determines whether the lookup must reside on the federated search head. The lookup on the federated search head is then used to generate results from the dataset returned from the remote deployment. If the search must run on the federated search head, but the lookup definition and lookup table aren't on the federated search head, Splunk Web displays a warning message letting the user know that the lookup needs to be processed on the federated search head, but the lookup definition and lookup table are missing; the lookup definition and lookup table need to be created on the federated search head. If the lookup doesn't exist on the remote search head, or the user doesn't have the correct permissions, then the remote provider will send the warning message to the local deployment, which will display it in Splunk Web.
The following table can help you anticipate where to place your lookup definitions and lookup tables.
| Location of lookup definition and lookup table file | Result |
|---|---|
| RSH only | Most simple standard mode federated searches with lookups complete without displaying a warning message. However, if the search performs complex aggregations, Splunk Web displays a warning message notifying the user that the lookup definition is missing from the local federated search head. |
| FSH only | The search fails and Splunk Web displays a warning message. |
| RSH and FSH | All standard mode federated searches complete successfully without displaying a warning message. |
Plan access to lookups
As an administrator, you need to decide how your users will use and access knowledge objects in your environment for the types of searches they will run. For example, if you don't want your users to run ad hoc searches against lookups, you should create saved searches for your users to run and set up lookups on your remote and federated search heads, as needed, for those saved searches. Alternatively, if you want your users to be able to run any kind of search, including ad hoc searches, you will need to let your users know which lookups and definitions are on which remote and federated search heads, so they can access them when they run their searches. Regardless of which approach you take, you should test your lookup command searches before deploying them to production to make sure that your lookup definitions and lookup tables exist on remote search heads and federated search heads as needed.
Configure your lookup to process on the federated search head of your local Splunk platform deployment
If you are running a standard mode federated search that uses the lookup command to enrich your results of a federated search, in some cases, you might want the lookup to be processed on the remote search head of the federated providers invoked in your search instead of the federated search head of your local Splunk platform deployment. Federated searches that process the lookup remotely have better overall search performance and standard mode federated searches that involve lookups complete faster on average when the lookup portion of the search is processed on the remote search heads of the federated providers invoked in the search. But, there might be reasons that you would prefer to have the lookup be processed on your federated search head on your local Splunk platform deployment.
If you are using standard mode federated search, and you want to process the lookup on your local federated search head, apply local=true to the search. When you apply local=true to a federated lookup search, the following things happen:
- The
lookupis processed on your local federated search head, using a lookup definition and lookup table that are located on that search head. - All commands following the
lookupare also processed on the local federated search head. - The portion of the search that precedes the
lookupcommand is processed on the remote search head of the federated provider.
local=true for lookup in a federated search, the local setting overrides the conditions that would cause the search to be processed on the remote search heads of the federated providers invoked in the search.
If you set up your federated search so that your local federated search head processes the lookup, the following conditions must be met for the search to return results.
- The lookup definition and lookup table expected by the
lookupcommand must exist on the federated search head. - The service account on remote search heads of your federated providers must have access permissions for the lookup definition and lookup table that are on the remote search heads. See Service accounts and security for Federated Search for Splunk.
See the lookup reference topic in the Search Reference.
Search head processing in standard mode federated searches with lookups
In the following examples, you run searches with lookup definitions located on local federated search heads and remote search heads.
1. Examples of lookups processed on the remote search head
The following example shows a simple search with a lookup that is processed only on the remote search head. This search requires the lookup definition and lookup table to exist only on the remote search head.
| search index=federated:remote_index
| lookup empAddress employeeIDThe following search with the lookup command also requires the lookup definition and lookup table to exist only on the remote search head.
| tstats prestats=true count from datamodel=federated:remote_internal_server by host
| tstats prestats=true append=true count from datamodel=federated:remote_internal_audit_logs by host
| lookup hosts.csv host
| stats count by host_description2. Examples of lookups processed on the federated search head
The following example of a search with a lookup is a more complex aggregation that Splunk software needs to process on the federated search head of the local Splunk platform deployment. As a result, duplicate lookup definitions and lookup tables must be present on both the local federated search head and the remote search head to avoid unexpected search results.
| mstats sum(_value) where index=federated:remote_index metric_name=* by host span=auto
| lookup hosts_1.csv hostThe following search with the lookup command also requires the lookup definition and lookup table on both the local federated search head and the remote search head.
|search index=remote_index1 OR index=federated:remote_index2
| lookup hosts_1.csv hostIf the lookup definition and lookup table aren't on the local federated search head that is responsible for processing the lookup, Splunk Web generates a warning message when the search runs.
Examples of different types of lookups in standard mode federated searches
Say you are using standard mode federated search, and you want to run a federated search that includes a custom CSV file-based lookup named empAddress. This lookup finds events in your search results with employeeID fields and adds corresponding address, city, country and postal_code field-value pairs to those events.
All CSV file-based lookups have two parts: a lookup definition, and a lookup table file. In this case, the lookup definition and lookup table file have the names empAddress and employee_addresses.csv, respectively.
For this example, you run three searches.
1. Example search with a lookup on a remote index on a federated provider
The following search with the lookup command applies the lookup to results from a remote index on the federated provider. Because this is a simple streaming search, the lookup definition and lookup table are only needed on the remote search head, and the remote indexers are responsible for applying the lookup on the results.
| search index=federated:remote_index
| lookup empAddress employeeID2. Example search with a lookup on a local federated search head
The following search with the lookup command aggregates results and must be processed on the local federated search head. As a result, the lookup definition and lookup table must be on both the remote search head and federated search head.
| mstats sum(_value) where index=federated:remote_index metric_name=* by host span=auto
| lookup empAddress employeeID3. Example search with a lookup on a local index and a remote index on a federated provider
The following streaming search with the lookup command applies the lookup to search results from an index on the local deployment, as well as a remote index on the federated provider. As a result, the lookup table and lookup definition need to be on both the remote search head and federated search head.
| search index=local_index OR index=federated:remote_index
| lookup empAddress employeeID