Route Heavy Forwarder data directly to Machine Data Lake

Configure a Heavy Forwarder input so processed events land in a Machine Data Lake raw table.

  • You have a Machine Data Lake raw table and know its resource name, such as ~.federated.<raw_table_name>.

  • The Heavy Forwarder already uses the existing Splunk Cloud receiver and TLS credentials in outputs.conf.

  • You can update the app that manages the input and restart the Heavy Forwarder.

Use this route when the Heavy Forwarder must parse, aggregate, filter, or otherwise process data before sending it to Splunk Cloud Platform. The input configuration selects the Machine Data Lake raw table destination. The existing outputs.conf configuration continues to send data to the Splunk Cloud receiver with the existing TLS credentials. There is no Machine Data Lake-specific outputs.conf stanza.

  1. Open the app that manages the input on the Heavy Forwarder.

    Put the input override in the app's local/inputs.conf file. For example, use $SPLUNK_HOME/etc/apps/<app_name>/local/inputs.conf.

  2. Add or update the input so the index target is the Machine Data Lake raw table resource name.
    CODE 
    [monitor:///var/log/cisco/asa.log]
disabled = false
sourcetype = cisco:asa
index = ~.federated.<raw_table_name>
  3. (Optional) Configure supported Heavy Forwarder processing.

    Use only supported props.conf and transforms.conf behavior for parsing, filtering, masking, source type assignment, or routing that must happen before the events are forwarded. Keep this processing separate from the Machine Data Lake raw table destination in the input stanza.

  4. Save the configuration files and restart the Heavy Forwarder.
    CODE 
    $SPLUNK_HOME/bin/splunk restart
  5. Generate a uniquely identifiable test event through the Heavy Forwarder input path.

    For example, send or write one event that includes mdl_test_id=<unique_id>, sourcetype="cisco:asa", and a recognizable action value.

  6. Verify the test event in the raw table.

    Open the raw table from the Catalog and search for the unique test value by using a narrow time range.

    CODE 
    mdl_test_id="<unique_id>" sourcetype="cisco:asa" source="/var/log/cisco/asa.log"

The Heavy Forwarder sends the events through the existing Splunk Cloud output connection, and Splunk Cloud Platform routes events with index = ~.federated.<raw_table_name> to the Machine Data Lake raw table.

For more information, see Deploy a heavy forwarder, Configure forwarding with outputs.conf, and Route and filter data.