Extended example: Deploy configurations to several forwarders

What we're aiming for

A common use for agent managements is to manage configuration files for forwarders. In some distributed environments, forwarders can number into the thousands, and the agent management greatly eases the work of configuring and updating them. This example shows how to use the agent management to initially configure a set of dissimilar universal forwarders. A follow-up example, in Example: Add inputs to forwarders, shows how to use the agent management to update the forwarders' configurations with new inputs.

The example sets up the following distributed environment, in which a agent management deploys configurations for three universal forwarders sending data to two indexers:

  • The agent management Fflanda-SPLUNK3 (10.1.2.4) manages deployment for these universal forwarders:
    • Fflanda-WIN1
    • Fflanda-LINUX1
    • Fflanda-LINUX2
  • Fflanda-WIN1 forwards Windows event logs to the receiving indexer Fflanda-SPLUNK1 (10.1.2.2).
  • Fflanda-LINUX1 and Fflanda-LINUX2 forward Linux messages to the receiving indexer Fflanda-SPLUNK2 (10.1.2.3).

The forwarders are agents, receiving their configuration files from agent management.

Here's the basic set up:

The forwarders are agents receiving their configuration files from agent management

For information on forwarders, see About forwarding and receiving in Forwarding Data. For more information about the universal forwarder, see About the universal forwarder in the Splunk Universal Forwarder product documentation.

For information on monitoring Windows event log data, see Monitor Windows event log data in Getting Data In.

For information on monitoring files, such as message logs, see Monitor files and directories in Getting Data In.

Configuration overview

Here's an overview of the set-up process (the detailed steps follow in the next section):

On each indexer that will be receiving data from the forwarders, do the following:

  1. Enable receiving through the CLI.
  2. Restart the indexer.

On each agent (forwarder), do the following:

  1. Create a deploymentclient.conf file that points to the agent management.
  2. Restart the forwarder.

On the agent management, do the following:

  1. Create directories for the deployment apps.
  2. Use agent management UI to create the set of server classes for the agents (forwarders). You'll create two server classes to represent the two OS types (Windows, Linux). Each server class will map a set of clients to two separate apps, for a total of four apps. These apps, defined in a later step, encapsulate:
    • The type of input -- the data that the universal forwarder will monitor (Windows event logs or Linux messages).
    • The type of output -- the indexer the forwarder will send data to (SPLUNK1 or SPLUNK2).
      This configuration results in each universal forwarder belonging to a server class and receiving two apps: one for its inputs and one for its outputs.
      Note: You can also create the server classes directly in serverclass.conf, as described below in Configure server classes in serverclass.conf
  3. Populate the app directories with the contents of the apps. Each app consists of a configuration file, either outputs.conf or inputs.conf.
  4. Deploy the apps by reloading the agent management.

After a short delay (while the forwarders receive and act upon their deployed content), Windows event logs begin flowing from Fflanda-WIN1 to Fflanda-SPLUNK1, and /var/log/messages begin flowing from Fflanda-LINUX1 and Fflanda-LINUX2 to Fflanda-SPLUNK2.

Detailed configuration steps

For each receiving indexer (Fflanda-SPLUNK1 and Fflanda-SPLUNK2):

  1. Install Splunk Enterprise on the machine where you want the indexer to reside.
  2. Run the following CLI command: 
    splunk enable listen 9997 -auth <username>:<password>
    This specifies that the indexer will serve as a receiver, listening for data on port 9997. With proper authorization, any forwarder can now send data to the receiver by designating its IP address and port number. You must enable the indexers as receivers before you configure the forwarders to send data to them.
    For more information on enabling receivers, see Enable a receiver in the Forwarding Data manual.
  3. Restart the indexer.

For each universal forwarder (Fflanda-WIN1, Fflanda-LINUX1, and Fflanda-LINUX2):

  1. Install the forwarder on the machine where you want it to reside, if you haven't already done so.
  2. Specify the agent management that the forwarder will poll. You can do this as part of the forwarder installation process, or later, by editing $SPLUNK_HOME/etc/system/local/deploymentclient.conf, like this: 
    [deployment-client]
   [target-broker:deploymentServer]
   # Specify the agent manager; for example, "10.1.2.4:8089".
   targetUri= <URI:port>
    See Configure agents for details on how to configure this file.
  3. Restart the forwarder.

On the agent management (Fflanda-SPLUNK3):

A. Prepare the agent management

  1. Install Splunk Enterprise, if you haven't already done so.
  2. Create these deployment app directories:
    • $SPLUNK_HOME/etc/deployment-apps/fwd_to_splunk1
    • $SPLUNK_HOME/etc/deployment-apps/fwd_to_splunk2
    • $SPLUNK_HOME/etc/deployment-apps/winevt
    • $SPLUNK_HOME/etc/deployment-apps/linmess

The directory name determines the name of the app: "fwd_to_splunk1", "fwd_to_splunk2", etc.

B. Configure the server classes for Windows forwarders

Note: This procedure and the one that follows both use agent management UI to configure the server classes. For an example of how to configure these server classes instead by direct editing of serverclass.conf, see [http://docs.splunk.com/Documentation/Splunk/10.0.0/Updating/Extendedexampledeployseveralstandardforwarders#Configure_server_classes_with_serverclass.conf Configure server classes in serverclass.conf] later in this topic.

To configure the Fflanda-WIN server class for Windows forwarders, go to the agent management interface and perform the following actions:

  1. Select the Server Classes tab.
  2. Select the New server class button.
  3. In the pop-up window, in the Server class name field, enter Fflanda-LINUX and select Save. The page with details of this server class opens.
  4. Select the Applications tab and then select the Edit applications button. This takes you to the Edit applications page. On this page, there are two sections: Unassigned Applications and Assigned Applications, with a list of apps under each. For a new server class, all apps will initially be in the Unassigned Applications section.
  5. In the Unassigned Applications section, look for the apps winevt and fwd_to_splunk1. Select both apps and use the arrow to move them to the Assigned Applications section.
  6. Select Save to save the apps in the server class.
  7. On the Server classes page for Fflanda-LINUX page go to the Forwarders tab and select Edit forwarders.
  8. On the Edit forwarders page you can add clients to the server class. To configure a filter to include all Windows clients, enter Fflanda-WIN* in the Include field and select Save.
  9. Set the apps' post-deployment behavior. For each app, go to the Applications tab, find the row with the app, and select this application to edit it. In the Settings section, select Enable application and Restart agent.

C. Configure the server classes for Linux forwarders

To configure the Fflanda-LINUX server class for Linux forwarders, go to the agent management interface and perform the following actions:

  1. Select the Server Classes tab.
  2. Select the New server class button.
  3. In the pop-up window, in the Server class name field, enter Fflanda-LINUX and select Save. This takes you to the page with details of this newly created server class.
  4. Select the Edit applications button. To find the button, you may need to select the arrow next to the Edit forwarders button.
  5. In the Unselected Applications column, look for the apps linmess and fwd_to_splunk2. Select each of these apps and use the right arrow to move them to the Selected Applications column.
  6. Select Save to save the apps in the server class.
  7. Go to the Forwarders tab and select Edit forwarders.
  8. On the Edit forwarders page, to configure a filter to include all Linux clients, enter Fflanda-LINUX* in the Include field and select Save.
  9. Set the apps' post-deployment behavior. For each application, go to the Applications tab, find the row with the application and select it. In the Settings section, select Enable application and Restart agent.

To learn more about the agent management interface, see Use agent management UI to define server classes.

D. Populate the apps

  1. Create $SPLUNK_HOME/etc/deployment-apps/fwd_to_splunk1/default/outputs.conf with the following settings: 
        [tcpout]
    defaultGroup=splunk1

    [tcpout:splunk1]
    # Specifies the server that receives data from the forwarder.
    server=10.1.2.2:9997
    For information on outputs.conf, see Configure forwarders with outputs.conf in the Forwarding Data manual.
  2. Create $SPLUNK_HOME/etc/deployment-apps/fwd_to_splunk2/default/outputs.conf with the following settings: 
        [tcpout]
    defaultGroup=splunk2

    [tcpout:splunk2]
    server=10.1.2.3:9997
  3. Create $SPLUNK_HOME/etc/deployment-apps/winevt/default/inputs.conf with the following settings: 
        [WinEventLog:Application]
    disabled=0

    [WinEventLog:Security]
    disabled=0

    [WinEventLog:System]
    disabled=0
    For information on monitoring Windows event log data, see Monitor Windows event log data in the Getting Data In manual.
  4. Create $SPLUNK_HOME/etc/deployment-apps/linmess/default/inputs.conf with the following settings: 
        [monitor:///var/log/messages]
    disabled=false
    sourcetype=syslog

For information on monitoring files, such as message logs, see Monitor files and directories in the Getting Data In manual.

E. Deploy the apps

To deploy the apps, just reload the agent management: 

splunk reload deploy-server

Each forwarder will now poll the agent management, download its configuration files, restart, and begin forwarding data to its receiving indexer.

Note: When you change the content of an app, you must always invoke the reload command, so that the clients know they need to download the new content.

For a follow-up example showing how to use the agent management to update forwarder configurations, see Example: Add inputs to forwarders.

Configure server classes with serverclass.conf

Instead of using agent management UI to configure server classes, you can directly edit serverclass.conf. For most purposes, however, it is faster and easier to use agent management UI.

In the case of the example under discussion, you can fully define the required server classes with agent management UI, so it is not necessary to edit serverclass.conf. However, for those who prefer to work directly in configuration files, this subtopic shows how to create the example's set of server classes by editing serverclass.conf. This version takes advantage of some minor optimization only available through direct editing of serverclass.conf.

Create $SPLUNK_HOME/etc/system/local/serverclass.conf with the following settings: 

    # Global server class 
    [global]
    # Filter (whitelist) all clients
    whitelist.0=*

    # Server class for Windows
    [serverClass:Fflanda-WIN]
    # Filter (whitelist) all Windows clients
    whitelist.0=Fflanda-WIN*

    # App for inputting Windows event logs
    # This app is only for clients in the server class Fflanda-WIN
    [serverClass:Fflanda-WIN:app:winevt]
    #Enable the app and restart Splunk, after the client receives the app
    stateOnClient=enabled
    restartSplunkd=true

    # App for forwarding to SPLUNK1
    # This app is only for clients in the server class Fflanda-WIN
    [serverClass:Fflanda-WIN:app:fwd_to_splunk1]
    stateOnClient=enabled
    restartSplunkd=true
    
    # Server class for Linux
    [serverClass:Fflanda-LINUX]
    # Filter (whitelist) all Linux clients
    whitelist.0=Fflanda-LINUX*
    
    # App for inputting Linux messages
    # This app is only for clients in the server class Fflanda-LINUX
    [serverClass:Fflanda-LINUX:app:linmess]
    stateOnClient=enabled
    restartSplunkd=true

    # App for forwarding to SPLUNK2
    # This app is only for clients in the server class Fflanda-LINUX
    [serverClass:Fflanda-LINUX:app:fwd_to_splunk2]
    stateOnClient=enabled
    restartSplunkd=true

See "Use serverclass.conf to define server classes" for details on how to configure this file.

What the communication between the agent management and its agents looks like

Using the above example, the communication from Fflanda-WIN1 to Fflanda-SPLUNK3 on port 8089 would look like this:

Fflanda-WIN1: Hello, I am Fflanda-WIN1.

Fflanda-SPLUNK3: Hello, Fflanda-WIN1. I have been expecting to hear from you. I have you down as a member of the Fflanda-WIN server class, and you should have the fwd_to_splunk1 (checksum=12345) and winevt (checksum=12378) apps.

Fflanda-WIN1: Hmmm, I don't have those configs. Using this connection I just opened up to you, can I grab the configs from you?

Fflanda-SPLUNK3: Sure! I have them ready for you.

Fflanda-WIN1: Thanks! I am going to back off a random number of seconds between 1 and 60 (in case you have a lot of agents that are polling you at the moment) ... OK, now send me the files.

Fflanda-SPLUNK3: Done! You now have fwd_to_splunk1-timestamp.bundle and winevt-timestamp.bundle.

Fflanda-WIN1: Awesome! I am going to store them in my $SPLUNK_HOME/etc/apps directory. Now I am going to restart myself, and when I come back up I am going to read the configurations that you sent me directly out of the .bundle files, which I know are just tar balls with a different extension.

A couple of minutes go by....

Fflanda-WIN1: Hello, I am Fflanda-WIN1.

Fflanda-SPLUNK3: Hello, Fflanda-WIN1. I have been expecting to hear from you. I have you down as a member of the Fflanda-WIN server class, and you should have the fwd_to_splunk1 (checksum=12345) and winevt (checksum=12378) Apps.

Fflanda-WIN1: Hmmm, I already have both of those, but thanks anyway!

Later on, an admin modifies the winevt/inputs.conf file on Fflanda-SPLUNK3 to disable the collection of system event logs, and then runs the CLI command splunk reload deploy-server to force the agent management to rescan serverclass.conf and the app directories. The next time Fflanda-WIN1 talks to Fflanda-SPLUNK3, it goes like this:

Fflanda-WIN1: Hello, I am Fflanda-WIN1.

Fflanda-SPLUNK3: Hello, Fflanda-WIN1. I have been expecting to hear from you. I have you down as a member of the Fflanda-WIN server class, and you should have the fwd_to_splunk1 (checksum=12345) and winevt (checksum=13299) Apps.

Fflanda-WIN1: Hmmm, I know I have those configs, but the checksum I have for the winevt configs is different than the one you just told me about. Using this connection I just opened up to you, can I grab the updated winevt config from you?

Fflanda-SPLUNK3: Sure! I have it ready for you.

Fflanda-WIN1: Thanks! I am going to back off a random number of seconds between 1 and 60 (in case you have a lot of agents that are polling you at the moment) ... Ok, now send me the updated config.

Fflanda-SPLUNK3: Done! You now have winevt-newer_timestamp.bundle.

Fflanda-WIN1: Awesome! I am going to store it in my $SPLUNK_HOME/etc/apps directory and move the old winevt.bundle I had out of the way. Now I am going to restart myself, and when I come back up, I'll have the most up-to-date config.