Manage knowledge objects for standard mode federated providers

Note: Skip this topic if you are using a transparent mode federated provider. When you use transparent mode federated search, the Splunk platform coordinates knowledge objects for you by using automatic bundle replication to bring your local knowledge objects to the remote search head.

If you plan to use Federated Search for Splunk to run federated searches that invoke your knowledge objects over a standard mode federated provider, identify the knowledge objects that you want to use in your searches and make sure they are present on the required search heads:

  • All knowledge objects that are used in a specific standard mode federated search must be defined on the remote deployment. This means that all knowledge objects in your search, such as calculated fields, event types, tags, and lookups must be present on the remote search head or the search will fail. For some commands, knowledge objects must also be defined on the local federated search head, especially if the knowledge object is needed on the local deployment to run the search.
  • Calculated fields, and, for some types of searches, definitions for lookups, event types, and tags must also be on the local federated search head. If this duplication of knowledge objects is not present as required, searches might fail or return errors. Splunk Web displays a warning message if a knowledge object is required on the federated search head, in addition to the remote search head.

Making knowledge objects available on the remote search heads, and federated search heads, as needed, helps ensure your federated searches complete without errors and return correct results. For example, if you are running a standard mode federated search that references a calculated field, the definition for the calculated field must be present on the local and remote sides of the federated search; if the calculated field doesn't exist on the remote search head, the remote search head can't apply the calculated field to search results from the federated provider, and if the calculated field doesn't exist on the federated search head, the search fails.

Administer knowledge object definitions for standard mode federated providers

Administrators with permissions to manage knowledge objects should have a plan for making knowledge object definitions available on remote and federated search heads, as needed, for the types of searches their users will run. Keep the following considerations in mind:

  • The knowledge objects from the remote deployment should be accessible to the federated provider service account defined on the remote search head.
  • Decide which knowledge objects should be available for your users and which should be restricted.
  • Make sure knowledge objects are defined on the remote and federated search heads that users need to access. This is especially important if your users don't have permission to add definitions to search heads themselves.
  • Test your knowledge object definitions before deploying them to production to make sure that they work as expected in typical searches your users will run.
  • If users must have access to knowledge object definitions on remote and federated search heads, give them a list of available knowledge objects and their corresponding remote provider locations.

Duplicate knowledge object names and definitions

When you prepare to run federated searches with knowledge objects over standard mode federated providers, you can arrange for your searches to run without knowledge object errors by ensuring that there are knowledge objects with the same names and definitions on the local and remote sides of the search. Improve the likelihood of getting correct results from a standard mode federated search that involves knowledge objects by duplicating the names and definitions of those knowledge objects and related files (such as CSV files, for CSV file lookups) on the local federated and remote search heads.

Note: If identically-named local and remote knowledge objects have different definitions, your searches might return results that use different values for the same search, which can lead to inconsistent results.

Ensure custom knowledge objects exist on remote and federated search heads

After you identify the custom knowledge objects that your users can use in their federated searches, make sure those knowledge objects are present on the remote search head on the federated provider and the federated search head, as needed. In most cases the easiest way to do this is through Splunk Web.

Prerequisites

  • Knowledge object verification requires admin access to the corresponding local and remote search heads where the knowledge objects are defined. If you do not have admin access to a Splunk platform deployment where you must duplicate knowledge objects, coordinate this work with the administrator of that deployment.
  • Learn about federated provider service accounts. See Service accounts and security for Federated Search for Splunk.

Steps

  1. Identify a knowledge object that you want to use in your federated searches.
  2. Verify that the knowledge object exists with identical definitions on the local and remote deployments involved in the search by looking it up in Settings on each deployment. See Help with knowledge objects.
  3. If the knowledge object does not exist on a deployment involved in the search, duplicate its definition on the deployment.
  4. Ensure that the remote instance of the knowledge object has its permissions set so that the federated provider service account can access it. See Manage knowledge object permissions in the Knowledge Manager Manual.
  5. If the knowledge object is a lookup, duplicate the lookup file or collection and upload or install it in the federated provider.

Repeat this process for each knowledge object you intend to use in your federated searches.

Use SPL commands with knowledge objects

Most commands are processed on both the federated and remote search heads in standard mode federated searches, and require relevant knowledge objects to also be defined on both search heads. However, the following commands are processed only on the federated search head and require relevant knowledge objects to be defined only on the federated search head. As a result, the knowledge object definitions used by these commands don't need to be located on the remote search head in order for searches to complete successfully:

Note: Your search might also need knowledge object definitions on a remote search head if the sendalert, mcollect, or outputlookup command is preceded in the search pipeline by commands that require relevant knowledge object definitions on the remote search head.

Run standard mode federated searches using event types and tags

Splunk handles event types and tags differently than lookups and other knowledge objects. Definitions for event types and tags must be on the remote search head, and depending on the search requirements, these knowledge object definitions might also be required on the federated search head. To avoid errors and ensure searches complete successfully, place knowledge object definitions on both remote search heads and federated search heads, if possible.

If the event type or tag definitions aren't present on the federated search head or remote search head as needed to successfully complete a search, Splunk Web displays a warning message indicating that they are missing. For example, if your search involves a remote dataset on a federated provider and a dataset on your local Splunk platform deployment, event types and tags must be defined on both the remote search head and the federated search head. If the knowledge objects are defined only on the remote search head, the federated search head warns the user that the definition doesn't exist on the federated search head.

You can use the presence or absence of a warning message in Splunk Web to guide you to determine whether you need to add an event type or tag definition to a remote search head or a local federated search head. The following table can help you anticipate where to place event type and tag definitions.

Location of event type or tag definition Description
RSH only The federated search head displays a warning message letting you know that the event type or tag definition is missing from the local federated search head. You can choose to add the definition to the federated search head or ignore the warning if the event type isn't required on the federated search head.

If the search involves only federated datasets on remote standard mode federated providers, then event types and tags only need to be defined on the remote search head; you don't need to define these knowledge objects on the local federated search head because the search is processed remotely.

FSH only The federated search head displays a warning message generated by the remote search head letting you know that the event type or tag definition is missing from the remote search head. You can choose to add the definition to the remote search head on the local deployment or ignore the warning if the event type isn't required on the remote search head.

If the search doesn't involve a remote dataset on a federated provider, the search completes successfully without errors. You don't need to include the event type and tag definitions on the remote search head because the search is just processed locally on the federated search head.

RSH and FSH All searches complete successfully without errors.

Note: Since search expansion for event types and tags are just text replacements, and Splunk platform only checks dataset types in the search and the search heads where the definitions are located, it doesn't matter where the event type or tag is placed in the search string. For example, the following search strings generate the same warning message regarding the location of the tag definition on search heads and the types of datasets used in the searches, even though the tags are defined for different datasets:
(index=idx1 tag=p1) OR index=federated:id2
index=idx1 OR (index=federated:id2 tag=p1)

For more information, see About event types and About tags and aliases.

Examples of federated searches with event types and tags

In the following examples, you run searches with event type and tag definitions located on local federated search heads and remote search heads.

1. Example search on a local index

The following search with the event type or tag is processed only on a local index, so the definition must be on the local federated search head: 

index=local_index eventtype=eventtype_a
index=local_index tag=tag_a

The following table shows you how the results for this search over a standard mode federated provider vary depending on where the event type or tag are defined. A warning message indicates that the event type or tag definition doesn't exist.

Location of the event type or tag definition Result
No definition on FSH or RSH The search completes with a warning message from the Search app.
RSH only The search completes with a warning message from the Search app.
FSH only The search completes without any warning messages.
RSH and FSH The search completes without any warning messages.

2. Example search on a remote index on a federated provider

The following search applies the event type or tag to results from a remote index on the federated provider, so the definition must be on remote search head: 

index=federated:remote_index eventtype=eventtype_a
index=federated:remote_index tag=tag_a

The following table shows you how the results for this search over a standard mode federated provider vary depending on where the event type or tag are defined. A warning message indicates that the event type or tag definition doesn't exist.

Location of the event type or tag definition Result
No definition on FSH or RSH The search completes with a warning message from the RSH.
RSH only The search completes without any warning messages.
FSH only The search completes with a warning message from the RSH.
RSH and FSH The search completes without any warning messages.

3. Example search on a local index and a remote index on a federated provider

The following search applies the event type or tag to search results from an index on the local deployment, as well as a remote index on the federated provider. As a result, the definition must be on both the local federated search head and the remote search head: 

index=local_index OR index=federated:remote_index eventtype=eventtype_a
index=local_index OR index=federated:remote_index tag=tag_a

The following table shows you how the results for this search over a standard mode federated provider vary depending on where the event type or tag are defined. A warning message indicates that the event type or tag definition doesn't exist.

Location of the event type or tag definition Result
No definition on FSH or RSH The search completes with a warning message from the FSH and another warning from the RSH.
RSH only The search completes with a warning message from the FSH.
FSH only The search completes with a warning message from the RSH.
RSH and FSH The search completes without any warning messages.

Run standard mode federated searches over lookups

For federated searches over standard mode that use the lookup command, Splunk software optimizes processing of the lookup on the federated search head of your local Splunk platform deployment and the remote search head of federated providers depending on the specific conditions of the search. For search performance reasons, Splunk software processes searches with the lookup command on the remote search heads of the federated providers when possible. Most simple searches with lookups are run only on remote search heads.

If you are running federated searches over standard mode Splunk platform federated providers, and you want to use the lookup command to enrich the results of a federated search, the following conditions must be met for the search to return results.

  • The lookup definition and lookup table expected by the lookup command must exist on the remote search heads. For some types of complex searches with the lookup command, the lookup definition and lookup table must also exist on the federated search heads.
  • The service account on remote search heads of your federated providers must have access permissions for the lookup definition and lookup table that are on the remote search heads. See Service accounts and security for Federated Search for Splunk.

For example, before a search that uses the lookup command and calculates aggregate statistics is sent to the remote search head, the local federated search head first determines whether the lookup must reside on the federated search head. The lookup on the federated search head is then used to generate results from the dataset returned from the remote deployment. If the search must run on the federated search head, but the lookup definition and lookup table aren't on the federated search head, Splunk Web displays a warning message letting the user know that the lookup needs to be processed on the federated search head, but the lookup definition and lookup table are missing; the lookup definition and lookup table need to be created on the federated search head. If the lookup doesn't exist on the remote search head, or the user doesn't have the correct permissions, then the remote provider will send the warning message to the local deployment, which will display it in Splunk Web.

The following table can help you anticipate where to place your lookup definitions and lookup tables.

Location of lookup definition and lookup table file Result
RSH only Most simple standard mode federated searches with lookups complete without displaying a warning message. However, if the search performs complex aggregations, Splunk Web displays a warning message notifying the user that the lookup definition is missing from the local federated search head.
FSH only The search fails and Splunk Web displays a warning message.
RSH and FSH All standard mode federated searches complete successfully without displaying a warning message.

Plan access to lookups

As an administrator, you need to decide how your users will use and access knowledge objects in your environment for the types of searches they will run. For example, if you don't want your users to run ad hoc searches against lookups, you should create saved searches for your users to run and set up lookups on your remote and federated search heads, as needed, for those saved searches. Alternatively, if you want your users to be able to run any kind of search, including ad hoc searches, you will need to let your users know which lookups and definitions are on which remote and federated search heads, so they can access them when they run their searches. Regardless of which approach you take, you should test your lookup command searches before deploying them to production to make sure that your lookup definitions and lookup tables exist on remote search heads and federated search heads as needed.

Configure your lookup to process on the federated search head of your local Splunk platform deployment

If you are running a standard mode federated search that uses the lookup command to enrich your results of a federated search, in some cases, you might want the lookup to be processed on the remote search head of the federated providers invoked in your search instead of the federated search head of your local Splunk platform deployment. Federated searches that process the lookup remotely have better overall search performance and standard mode federated searches that involve lookups complete faster on average when the lookup portion of the search is processed on the remote search heads of the federated providers invoked in the search. But, there might be reasons that you would prefer to have the lookup be processed on your federated search head on your local Splunk platform deployment.

If you are using standard mode federated search, and you want to process the lookup on your local federated search head, apply local=true to the search. When you apply local=true to a federated lookup search, the following things happen:

  • The lookup is processed on your local federated search head, using a lookup definition and lookup table that are located on that search head.
  • All commands following the lookup are also processed on the local federated search head.
  • The portion of the search that precedes the lookup command is processed on the remote search head of the federated provider.

Note: When you set local=true for lookup in a federated search, the local setting overrides the conditions that would cause the search to be processed on the remote search heads of the federated providers invoked in the search.

If you set up your federated search so that your local federated search head processes the lookup, the following conditions must be met for the search to return results.

  • The lookup definition and lookup table expected by the lookup command must exist on the federated search head.
  • The service account on remote search heads of your federated providers must have access permissions for the lookup definition and lookup table that are on the remote search heads. See Service accounts and security for Federated Search for Splunk.

See the lookup reference topic in the Search Reference.

Search head processing in standard mode federated searches with lookups

In the following examples, you run searches with lookup definitions located on local federated search heads and remote search heads.

1. Examples of lookups processed on the remote search head

The following example shows a simple search with a lookup that is processed only on the remote search head. This search requires the lookup definition and lookup table to exist only on the remote search head. 

| search index=federated:remote_index
| lookup empAddress employeeID

The following search with the lookup command also requires the lookup definition and lookup table to exist only on the remote search head. 

| tstats prestats=true count from datamodel=federated:remote_internal_server by host 
| tstats prestats=true append=true count from datamodel=federated:remote_internal_audit_logs by host 
| lookup hosts.csv host 
| stats count by host_description

2. Examples of lookups processed on the federated search head

The following example of a search with a lookup is a more complex aggregation that Splunk software needs to process on the federated search head of the local Splunk platform deployment. As a result, duplicate lookup definitions and lookup tables must be present on both the local federated search head and the remote search head to avoid unexpected search results. 

| mstats sum(_value) where index=federated:remote_index metric_name=* by host span=auto
| lookup hosts_1.csv host

The following search with the lookup command also requires the lookup definition and lookup table on both the local federated search head and the remote search head. 

|search index=remote_index1 OR index=federated:remote_index2
| lookup hosts_1.csv host

If the lookup definition and lookup table aren't on the local federated search head that is responsible for processing the lookup, Splunk Web generates a warning message when the search runs.

Examples of different types of lookups in standard mode federated searches

Say you are using standard mode federated search, and you want to run a federated search that includes a custom CSV file-based lookup named empAddress. This lookup finds events in your search results with employeeID fields and adds corresponding address, city, country and postal_code field-value pairs to those events.

All CSV file-based lookups have two parts: a lookup definition, and a lookup table file. In this case, the lookup definition and lookup table file have the names empAddress and employee_addresses.csv, respectively.

For this example, you run three searches.

1. Example search with a lookup on a remote index on a federated provider

The following search with the lookup command applies the lookup to results from a remote index on the federated provider. Because this is a simple streaming search, the lookup definition and lookup table are only needed on the remote search head, and the remote indexers are responsible for applying the lookup on the results. 

| search index=federated:remote_index
| lookup empAddress employeeID

2. Example search with a lookup on a local federated search head

The following search with the lookup command aggregates results and must be processed on the local federated search head. As a result, the lookup definition and lookup table must be on both the remote search head and federated search head. 

| mstats sum(_value) where index=federated:remote_index metric_name=* by host span=auto
| lookup empAddress employeeID

3. Example search with a lookup on a local index and a remote index on a federated provider

The following streaming search with the lookup command applies the lookup to search results from an index on the local deployment, as well as a remote index on the federated provider. As a result, the lookup table and lookup definition need to be on both the remote search head and federated search head. 

| search index=local_index OR index=federated:remote_index
| lookup empAddress employeeID

Help with knowledge objects

The following table lists knowledge object definitions, files, and collections that might need to be duplicated on your federated and remote search heads if you want to use them in federated searches. You can verify the existence of a knowledge object by looking it up in Settings for your local deployment and the remote deployments involved in the federated search.

All links go to topics in the Knowledge Manager Manual unless otherwise indicated.

Type of knowledge object Items that might need to be duplicated among the federated and remote search heads For more information
Custom search-time field extraction Field extraction configurations About fields
Calculated field Calculated field definition About calculated fields
Field alias Field alias definition Create field aliases in Splunk Web
CSV file lookup
  • Lookup definition
  • CSV lookup table file
 Define a CSV lookup in Splunk Web
External lookup
  • Lookup definition
  • External lookup script
 Create external lookups for apps in Splunk Cloud Platform or Splunk Enterprise in the Developer Guide on the Developer Portal
KV Store lookup
  • Lookup definition
  • KV store collection
 Define a KV store lookup in Splunk Web
Geospatial lookup
  • Lookup definition
  • The .kmz or .kml lookup table file
 Define a geospatial lookup in Splunk Web
Event type Event type definition About event types
Search macro Search macro definition Define search macros in Settings
Tag Tag definition Define and manage tags in Settings