Steps to disable rule:

1. remove (or move to some other location outside of UBA as a backup) the file /etc/caspida/conf/rules/user/ad/augmented_access.rule

2. sync-cluster (/opt/caspida/bin/Caspida sync-cluster /etc/caspida/conf/rules/user/ad/)

3. restart uba (/opt/caspida/bin/Caspida stop & /opt/caspida/bin/Caspida start)

1. Remove go.mod from UBA upgrade and/or install packages.

2. Upgrade node.js to 20.9.0.

/opt/caspida/bin/Caspida stop-all

b. Remove pre-existing node.js files on all UBA nodes:

sudo rm -rvf /usr/local/bin/corepack
sudo rm -rvf /usr/local/bin/node
sudo rm -rvf /usr/local/bin/npm
sudo rm -rvf /usr/local/bin/npx
sudo rm -rvf /usr/local/share/systemtap/tapset/node.stp
sudo rm -rvf /usr/local/share/doc/node
sudo rm -rvf /usr/local/share/man/man1/node.1
sudo rm -rvf /usr/local/lib/node_modules/
sudo rm -rvf /usr/local/include/node/

wget https://nodejs.org/dist/v20.9.0/node-v20.9.0-linux-x64.tar.gz

sudo tar xvfz node-v20.9.0-linux-x64.tar.gz -C /usr/local --strip-components 1

e. Verify node and npm versions on all UBA nodes:

node -v
--- 20.9.0
npm -v
--- 10.1.0

/opt/caspida/bin/Caspida start-all

sed -i 's/service postgresql \*,/service postgresql\*,/g' /opt/caspida/etc/sudoers.d/ubasudoers

Error:

	Tue Oct 17 16:37:04 UTC 2023: adding worker node: <uba-hostname>
	Tue Oct 17 16:37:04 UTC 2023: joining worker: <uba-hostname>, cmd=sudo  kubeadm join <uba-ip>:6443 --token <token> --discovery-token-ca-cert-hash <hash>  --cri-socket=unix:///var/run/cri-dockerd.sock --ignore-preflight-errors Swap
	Tue Oct 17 16:42:04 UTC 2023: Failed to add worker: uba-hostname2.local, exiting

Steps: a) Manually generate the token from the management node:

sudo kubeadm token create --print-join-command

b) Run the command from the second node with sudo and the necessary flags:

sudo <output of the fist command> --cri-socket=unix:///var/run/cri-dockerd.sock --ignore-preflight-errors Swap --v=5

c) Here, if you observe the error is due to "dial tcp <uba-ip>:6443: connect: no route to host."

d) Open the 6443 port on all worker nodes:

sudo firewall-cmd --permanent --zone=public --add-port=6443/tcp
	    sudo firewall-cmd --reload

e) Re-run the Caspida setup.

2) If you have not attached the second interface and have not run the Caspida setup yet, follow the steps below:

a) Build a server with a single interface.

b) Ensure that the interface is associated with the public firewall zone.

c) Install Splunk UBA following Splunk documentation using the node names that resolve to an IP address on this attached interface.

d) Perform post-installation sync.

e) Stop all UBA services.

f) Create a new firewall zone named "control-plane".

g) Add a new interface associated with the "control plane" zone.

h) Add inbound firewall rules permitting SSH and HTTPS.

i) Start all UBA services.

Run the following query from the management node (for 1, 3, 5, 7, and 10 node deployment) and 2nd node (for 20 and 20XL node deployment):

psql -d caspidadb -c "UPDATE rulepackages SET version = 0 WHERE namespace LIKE 'secteam.%';"

From the management node, run the following command:

curl -w "\n" -Ssk -H "Authorization: Bearer $(grep '^\s*jobmanager.restServer.auth.user.token=' /opt/caspida/conf/uba-default.properties | cut -d'=' -f2)" -m 60 -X "POST" "https://localhost:9002/rules/installSystemPackages"

Note: There is NO need to restart any UBA services

They can safely resolve this by removing the entire /var/vcap/store/docker/aufs/diff/<Image Layer ID> directory since it does not correspond to any Docker image/container/volume currently used in UBA 5.3.0.

sudo sed -i 's/^supervised no$/supervised auto/g' /etc/redis/redis.conf

2. Take a backup of the ContainerizationCleanup.sh script:

cp /opt/caspida/containerization/bin/ContainerizationCleanup.sh /opt/caspida/containerization/bin/ContainerizationCleanup.sh.ORIG

3. Insert the following code snippet at the end of the ContainerizationCleanup.sh script before the last line: echo "$(date): $0: DONE"

#############
for node in "${!uniqHostsArr[@]}"
do
  echo "$(date): Check broken symlink of pods-log at Container Node : ${node}"
  ssh ${node} "(
    for i in \$(sudo find /var/log/pods -type l)
    do
      LOGFILE=\$(sudo ls -l \${i} | awk -F '->' '{print \$2}');
      if sudo test -f \$LOGFILE; then
         echo "$(date): INFO - symlink of pod-log is OK. symlink=\${i}";
      else
         echo "$(date): WARN - symlink of pod-log is BROKEN, so deleting it. symlink=\${i}";
         sudo rm \${i};
      fi
    done
  )"
done
#############

4. Run the following sync-cluster command:

/opt/caspida/bin/Caspida sync-cluster /opt/caspida/containerization/bin/

locate leader-epoch-checkpoint

a) Copy result into a script, adding ">" prior to each result. i.e.

#!/bin/bash
> /var/vcap/store/kafka/AnalyticsTopic-0/leader-epoch-checkpoint
> /var/vcap/store/kafka/AnalyticsTopic-1/leader-epoch-checkpoint
> /var/vcap/store/kafka/AnalyticsTopic-10/leader-epoch-checkpoint
> /var/vcap/store/kafka/AnalyticsTopic-11/leader-epoch-checkpoint
...

chmod +x <script name>.sh

/opt/caspida/bin/Caspida stop-all

./<script name>.sh

/opt/caspida/bin/Caspida start-all

tail -f /var/vcap/sys/log/kafka/server.log

6) If you see the following warning repeated:

WARN Resetting first dirty offset of __consumer_offsets-17 to log start offset 3346 since the checkpointed offset 3332 is invalid. (kafka.log.LogCleanerManager$)

> /var/vcap/store/kafka/cleaner-offset-checkpoint

/opt/caspida/bin/Caspida stop-all && /opt/caspida/bin/Caspida start-all

You can check deployments.conf field spark.history to find out which node runs the Spark History Server.

Update the following setting to 3G: SPARK_DAEMON_MEMORY=3G

Afterwards, restart the spark services:

/opt/caspida/bin/Caspida stop-spark && /opt/caspida/bin/Caspida start-spark

/opt/caspida/bin/Caspida remove-containerization
/opt/caspida/bin/Caspida setup-containerization
/opt/caspida/bin/Caspida stop-all
/opt/caspida/bin/Caspida start-all

| fields app,change_type,dest_user,file_hash,file_size,object,object_path,object_type,parent_category,parent_hash,sourcetype,src_user,tag,src_ip