Integrate a universal forwarder onto a Linux system image

To successfully integrate the universal forwarder onto a Linux system image, follow these best practices:

Start by deploying a small number of cloned systems.
The first time the universal forwarder starts in the cloned environment, ensure it uses unique GUIDs and the hostname. It prevents multiple universal forwarders from appearing as the same instance in the Splunk environment.
Customize the integration process to suit specific requirements of your company and document it.
Every time you upgrade the universal forwarder, test the integration process for each version.

Critical considerations

Prepare the universal forwarder for cloning properly because it ensures that the source and target systems use unique identifiers, such as GUIDs and hostnames. See Prepare the universal forwarder installation for cloning.

Note:
In Splunk Enterprise versions 8.2 and higher, verify that running the clone-prep-clear-config command cleared the hostname from the inputs.conf file. If it failed to clear it, manually remove the inputs.conf file by running this command:

rm /opt/splunkforwarder/etc/system/local/inputs.conf
Ensure cloned systems maintain proper file permissions. If you run the universal forwarder as a non-root user, ensure proper permissions by running this command:
```
sudo chown -R splunkfwd:splunkfwd /opt/splunkforwarder/
```
Ensure that the firewall settings on the cloned system allow the universal forwarder to connect to indexers.
Verify that the settings of token-based authentication are preserved during cloning.
Secure Sockets Layer (SSL) certificates that are tied to the hostname may require regeneration.

To install the universal forwarder on Linux, follow these steps:

Extract the universal forwarder package for the Linux system to the opt directory by running the following command:
```
tar -xzf splunkforwarder-*.tgz -C /opt/
```
Change the opt directory to the binary folder of the universal forwarder:
```
cd /opt/splunkforwarder/bin
```
Start Splunk Enterprise accepting the license automatically:
```
./splunk start --accept-license
```

The universal forwarder is installed on Linux.

Configure forwarding and monitoring on Linux by taking these steps:
1. Add the indexer:
```
./splunk add forward-server indexer.company.com:9997 
```
2. Configure data inputs to monitor logs:
```
./splunk add monitor /var/log/
```
Configure the universal forwarder to start automatically when Linux boots:
```
/opt/splunkforwarder/bin/splunk enable boot-start
```
To test the configuration of the universal forwarder on Linux, take these steps:
1. Verify the list of servers that the universal forwarder is forwarding data to.
```
/opt/splunkforwarder/bin/splunk list forward-server
```
2. Verify the list of files and directories that the universal forwarder is currently monitoring for data input.
```
/opt/splunkforwarder/bin/splunk list monitor
```

The universal forwarder is configured on Linux.

To prepare the universal forwarder installation for cloning, take the following steps:

The universal forwarder installation is ready for cloning.

To clone and deploy the Linux system image to target systems, follow these steps:

(Optional) To reduce the image size, clear unnecessary data from system logs by running the following commands:
- To clear general system log files:
```
sudo truncate -s 0 /var/log/*.log
```
- To clear log files in the log directory of the universal forwarder:
```
sudo truncate -s 0 /opt/splunkforwarder/var/log/splunk/*.log 
```
Clear the command history for the current user in the current shell session:
```
history -c
```
Create the Linux system image that captures the entire universal forwarder directory.
Use an imaging tool preferred by your organization.
Note: Do not reboot the source system before you create the image.
Deploy the image to target machines using standard deployment procedures in your organization.

Note: Ensure that each target system has a unique hostname and IP address.
Verify the deployment of the universal forwarder on the cloned system by taking these substeps:
1. Start the universal forwarder by running one of the following commands:
  The first time the universal forwarder starts in the cloned environment, it regenerates unique identifiers.
```
sudo systemctl start Splunkd 
```
  or
```
/opt/splunkforwarder/bin/splunk start
```
2. Verify that the cloneprep file does not exist:
```
ls -la /opt/splunkforwarder/cloneprep 
```
3. Check that unique identifiers are regenerated:
```
grep -i "serverName\|guid\|host" /opt/splunkforwarder/etc/system/local/*.conf 
```

The cloned image of the Linux system is deployed to the target system.