Run federated searches over remote Splunk platform deployments

Federated Search for Splunk lets you search across specific datasets on multiple Splunk platform deployments. These deployments include your local Splunk platform deployment and remote Splunk platform deployments that you designate as federated providers. A federated search is processed partly on your local instance and partly on the federated providers. The remote and local results are then brought to the federated search head of your local deployment, where they are aggregated together to produce a final result set.

The federated search process includes search optimizations that help federated searches run as efficiently as possible. These optimizations filter results on the remote search heads, which reduces the amount of data that must be sent over the network, and then process the results on your federated search head, improving overall search performance.

The experience of writing and running federated searches differs depending on whether your federated providers use standard mode or transparent mode.

For an overview of federated search and federated search terminology, see About Federated Search for Splunk.

General requirements for federated search

Before you can run a federated search, your administrator must designate one or more remote deployments as federated providers. See Define a Splunk platform federated provider. Find out whether the providers you are searching over use standard mode or transparent mode.

Federated provider names added to search results

When you run federated searches, the names of the federated providers you search are added to your search results as the values of a field named splunk_federated_provider. You can use the splunk_federated_provider field to group or filter results by the federated providers that produced them.

For example, if you run transparent mode federated searches over multiple federated providers, you can use the splunk_federated_provider field to discern how many results are coming from each provider. Without this field it can be hard to tell where your results are coming from. The following transparent mode federated search breaks out result counts by provider name and then again by product name.

sourcetype=access_* | stats count(eval(splunk_federated_provider="ProviderWest")) AS Provider_West, count(eval(splunk_federated_provider="ProviderEast")) AS Provider_East BY productName

This feature applies to all federated searches, whether they are in standard mode or transparent mode.

This feature requires that your local Splunk platform deployment and all remote deployments set up as federated providers be upgraded to the following Splunk platform versions:

Splunk Cloud Platform 9.0.2305 or higher.
Splunk Enterprise 9.0.6 or higher Splunk Enterprise 9.0.x versions.
Splunk Enterprise 9.1.1 or higher.

Search over a standard mode federated provider

Federated searches that run over a standard mode federated provider must use a special search syntax to invoke the federated search. When you work with standard mode federated providers and you run a search that does not include this syntax, the search runs only over your local deployment.

Standard mode federated searches are subject to specific requirements and restrictions.

Additional requirements for standard mode federated search

Your administrator must create one or more federated indexes on your local federated search head. Each federated index maps to a specific remote dataset on a federated provider. See Map a federated index to a remote Splunk dataset.
Your role must have permissions for the federated indexes on your local federated search head that you intend to search. See Give your users role-based access control of federated indexes.
All custom knowledge objects in your search, such as calculated fields, event types, tags, and lookups must be present on the remote search head. Calculated fields, and, for some types of searches, lookups, event types, and tags must also be defined on the local federated search head. If this duplication of knowledge objects is not present as required, searches might fail or return errors. See Manage knowledge objects for standard mode federated providers.

Write standard mode federated searches

The basic syntax for a standard mode federated search differs depending on the type of remote dataset that you are referencing in the search. A federated search of an events index dataset requires different syntax than a federated search of a metrics index, saved search, last job, or data model dataset.

However, all standard mode federated searches require a reference to at least one federated index that you have defined on your federated search head. This federated index maps to a remote dataset on the federated provider such as an events index, a metrics index, a saved search, a data model, or the last job run by a scheduled search. See Map a federated index to a remote Splunk dataset.

You can use Boolean operators such as AND and OR to reference multiple federated indexes in a subsearch.

If your role has the admin_all_objects and indexes_edit capabilities, you can view the federated indexes to which you have access and the remote datasets that those federated indexes map to on the Federated Indexes listing page at Settings > Federated Search > Federated Indexes. If your role does not have this capability, get the names of the federated indexes that you can search from your administrator.

Syntax for standard mode federated searches of remote datasets

The following table provides the search commands and syntax required to search various dataset types on a standard mode federated provider. All of the examples use the federated: prefix to invoke a federated index on the federated search head that maps to a dataset on the remote search head.

Note: When you run a standard mode federated search, you must follow the syntax for the dataset type to which your federated index is mapped. For example, you cannot use the from command to reference an events index, and you cannot search a metrics index without using mstats or mcatalog and referencing the federated index to which the federated index is mapped with a WHERE clause. When your search syntax does not match the dataset type, Splunk software returns an error message.


Remote dataset type	Required syntax	More information
events index	Use `search` to search events index datasets. `\| search index=federated:<federated_index_for_events_index>` or `index=federated:<federated_index_for_events_index>`	You can use a wildcard symbol (``) to reference all federated indexes that map to events indexes: `\| search index=federated:`
metrics index	Use `mstats` or `mcatalog` in conjunction with a WHERE clause to search metrics index datasets. `\| mstats <stats-func> WHERE index=federated:<federated_index_for_metrics_index>`	You cannot use a wildcard symbol (`*`) to reference all federated indexes that map to metrics indexes.
saved search	Use `from` or `savedsearch` to reference federated indexes that map to saved search datasets. `\| from federated:<federated_index_for_saved_search>` or `\| savedsearch federated:<federated_index_for_saved_search>`	A saved search dataset is composed of the results of an ad-hoc run of a saved search on the remote search head. For a comparison of saved search and last job datasets, see Map a federated index to a remote Splunk dataset.
last job	Use `from` to reference federated indexes that map to saved search datasets. `\| from federated:<federated_index_for_last_job_dataset>`	A last job dataset is composed of the results of the last job run by a scheduled search on the remote search head. For a comparison of saved search and last job datasets, see Map a federated index to a remote Splunk dataset.
data model	Use `from` to search an unaccelerated remote data model dataset. `\| from federated:<federated_index_for_data_model>` Use `tstats` in conjunction with a `FROM` clause to search an accelerated remote data model dataset. `\| tstats <stats-func> FROM datamodel=federated:<federated_index_for_data_model>`

Restrictions for standard mode federated search

Standard mode federated search does not support the following things:

Generating commands, with the exception of search, eventcount, from, loadjob, mcatalog, mstats, savedsearch, or tstats. For example, standard mode federated searches cannot include the datamodel or inputlookup commands. You can find a list of generating commands in Command types, in the Search Reference.
Using from to reference events index or metrics index datasets.
Real-time search.

Metrics-specific search commands, with the exception of mcollect, mstats and mcatalog. If you must use these commands in a federated search, consider mapping a federated index to a saved search or last job dataset that runs them. See Map a federated index to a remote Splunk dataset.

SPL commands that run on the federated search head in standard mode

Most SPL commands are processed on both the federated and remote search heads in standard mode searches, and require knowledge objects to be defined on both search heads. The following commands are processed only on the federated search head and don't require any relevant knowledge objects to be defined on the remote search head in order for searches to complete successfully:

The sendemail command, which doesn't use knowledge objects, is also processed only on the federated search head in standard mode federated searches.

See Manage knowledge objects for standard mode federated providers for more information about using knowledge objects in standard mode federated searches.

Standard mode federated search examples

Here are examples of federated searches of remote datasets on standard mode federated providers.

Simple search of a large remote events index dataset

Say you have a federated index on your local search head named airline_flights_SF. This federated index is mapped to an events index on a remote deployment that is defined as a federated provider for your local deployment. The remote events index contains 2,000,000 events relating to airline flight departures and arrivals at a San Francisco airport. To return the first 100 events of this large remote dataset, run this search:

index = federated:airline_flights_SF | head 100

Note: As a best practice, run a simple search like this on a federated index that you're unfamiliar with. This practice helps you determine whether the federated index name is valid, since you won't see data if the federated index is invalid. It also helps you understand whether the dataset that the federated index is mapped to holds data that is worth searching or correlating with information on your local deployment.

Simple search of a remote metrics dataset

Following on from the previous example, you might also have a federated index on your local search head named airline_flights_SF_metrics. This federated index is mapped to a metrics index on a remote deployment that is defined as a federated provider for your local deployment. This remote metrics index contains a significantly large number of metric data points that also relate to San Francisco airline flight arrivals and departures.

The following search returns a count of a specific airline departure metric.

| mstats count(SF.airline.splunkpony.departure_delay_time) where index=federated:airline_flights_SF_metrics

For more information, see mstats in the Search Reference and Overview of metrics in Metrics.

Simple search of a last job dataset

The following search runs over the last job of a scheduled search. The scheduled search runs at the start of each hour and returns information about security badge errors at gates for the preceding hour.

| from federated:badge_errors_last_hour | timechart span=1m count by gate usenull=f

Combine local and standard mode federated search results for comparative analysis

Say you want to investigate the performance of specific airline carriers at competing airports. The following search uses the union command to combine a remote saved search dataset of San Francisco flight data and a local events index dataset of New York City flight data. Then it finds the average arrival delay between airports in the combined dataset, broken out by airline.

| union 
  [savedsearch federated:search_of_airline_flights_SF] 
  [search index = airlinedata_NYC] 
| stats avg (ArrDelay) by UniqueCarrier

If you want to search a local events index and a remote events index, you can use an OR operator to combine the results:

index = airlinedata_NYC OR index = federated:airline_flights_SF
| stats avg (ArrDelay) by UniqueCarrier

Aggregate results with join for a count of unique values

The following standard mode federated search uses the join command to aggregate results from the remote and local deployments. It returns the total count of airline carriers running flights out of San Francisco and New York City.

index = federated:airline_flights_SF 
| join left = L right = R where L.UniqueCarrier = R.UniqueCarrier 
  [search index = airlinedata_NYC] 
| stats count

Filter a remote data model dataset search by one or more index names

When you run a tstats search of a remote data model dataset on a standard mode federated provider, it is often helpful to filter the results of the search by one or more indexes. Indexes referenced by the WHERE keyword must be present on the remote search head.

| tstats count from datamodel = federated:remote_dm_1
WHERE index=index_fs_1 OR index=index_fs_2

You can also run tstats searches that mix local and federated indexes.

|tstats count where index=local_index_1 OR index=federated:remote_dm_1

Filter a remote data model dataset search on child data model datasets

When you run a tstats search of a remote data model dataset on a standard mode federated provider, you can filter the results on one or more of the child datasets within the data model by referencing the nodename of the child dataset.

| tstats count from datamodel = federated:remote_dm_1
WHERE nodename = DM_1_DS.DM_1_DS_CHILD_1

See tstats in the Search Reference for more information about using nodename to filter tstats searches on child data model datasets.

Search over a transparent mode federated provider

When you run searches over a transparent mode federated provider, you can search as if you were searching over your local deployment. No special syntax is required. The search permissions associated with your role govern what you can search on the federated provider.

Searching accelerated data models in transparent mode

After you set up transparent mode federated search, accelerated data models on your local Splunk platform deployment create data model acceleration summaries on indexers of your local deployment and on indexers of your transparent mode federated provider. You do not need to set up accelerated data models on your transparent mode federated provider.

In your federated search, reference a local accelerated data model to return both local and remote results.

Note: The ability to run transparent mode federated searches over accelerated data models requires that both your local and remote Splunk platform deployments be at either Splunk Cloud Platform 9.0.2303 or higher, or Splunk Enterprise 9.1.0 or higher.

See About data models and Accelerate data models in the Knowledge Manager Manual.

Restrictions for transparent mode federated search

Transparent mode federated search does not support the following things:

Real-time search.
Using the meventcollect command in searches.
Using the datamodel command to search remote data models.
Using from to search saved search datasets on the federated provider. You can use from to search saved search datasets on your local deployment.
Using the federated: syntax to refer to specific federated indexes. Transparent mode searches do not use federated indexes.

Blocked commands for transparent mode federated search

Certain commands are blocked in transparent mode federated searches. When you use a blocked command in a transparent mode federated search, the search returns a warning message and returns results only from your local search head.

The following commands are entirely blocked in transparent mode federated searches:

delete
dump
map
rest
run
runshellscript
script
sendalert
sendemail.

The commands in this list are classified as risky commands. See SPL safeguards for risky commands in Securing Splunk.

In addition, in certain situations, the makeresults and tstats commands are blocked or restricted.

Transparent mode federated searches that use the makeresults command are blocked unless you use the splunk_server or splunk_server_group argument to reference a search head, search head cluster, indexer, or indexer cluster on either your local Splunk platform deployment or the remote federated provider involved in the search. Splunk software processes the makeresults search on the local or remote server that you indicate with either of these arguments. For more information see makeresults in the Search Reference.

tstats searches that include a FROM clause are blocked for transparent mode federated searches over federated providers with Splunk Cloud Platform versions lower than 9.0.2303 or Splunk Enterprise versions lower than 9.1.0. If you use multiple transparent mode federated providers, the tstats search is processed only on federated providers with qualifying versions. For more information see tstats in the Search Reference.

Documentation