Manage your PostgreSQL database with data retention strategies

Use data retention strategies to schedule and cleanup records in your Splunk SOAR (On-premises) PostgreSQL database.

Use configure_db_maintenance to set parameters for retaining or removing database entries in the Splunk SOAR (On-premises) PostgreSQL database.

A set of options is called a strategy.
Models are collected into groups.
Strategies are applied to groups.

Strategy

The set of configurable parameters that define when a record should be deleted, either automatically or when the db_maintenance tool runs.

Model

Any PostgreSQL database record or Django object is called a model. Models have characteristics that define what sort of information the model represents.


Group name	Description
`audit`	Audit logs. See Enable and download audit trail logs in Splunk SOAR (On-premises).
`automation_run`	Automation run data including Playbook Run, Action Run, App Run, Custom Function Run, and playbook run statistics.
`container`	Containers and related records like Artifacts and Container Attachments. See About Splunk SOAR (On-premises).
`indicator`	Indicators or Indicators of Compromise. See About Splunk SOAR (On-premises).
`playbook_run_log`	Debug and error logs from historical playbook runs.

To use the configure_db_maintenance.py tool, follow these steps:

SSH to your Splunk SOAR (On-premises) instance.
```
 SSH <username>@<SOAR_hostname> 
```
Use the following tool to manage data deletion.
```
 phenv configure_db_maintenance 
```
Add your desired argument to the data retention tool command line to list, add, edit or schedule data retention actions.

Note: On clustered systems, the configure_db_maintenance.py tool can be run from any node, but only the leader node runs the data retention strategy.

Data retention tool arguments

Add the --help argument to your command to get information on the data retention tool arguments;

  phenv configure_db_maintenance --help

Optional arguments

Use these optional arguments to manage your data retention strategy.


Argument	Description
`-h, --help`	Show this help message and exit.
`-v {0,1,2,3}, --verbosity {0,1,2,3}`	Set the verbosity for the command's output. 0 for minimal output 1 for normal output 2 for verbose output 3 for very verbose output
`--no-color`	Do not use color in the command output
`--skip-checks`	Do not perform system checks.


Subcommand	Description
`list`	List all existing data retention strategies and their schedules.
`add`	Add a new data retention strategy, or update a data retention strategy by updating all of that strategy's settings. This command has options. See the table for add later in this topic.
`edit`	Selectively update an existing data retention strategy. This command has options. See the table for edit later in this topic.
`schedule`	Add or update a scheduled job to service all data retention strategies. This command has options. See the table for schedule later in this topic.

configure_db_maintenance subcommands

Use configure_db_maintenance to list, add, edit or schedule data retention strategies.

List

Get a list of all data retention strategies by using the list subcommand.

phenv configure_db_maintenance list

Sample output

Enabled data retention strategies will be serviced on the following cron schedule: Every hour UTC (cron string: "0 * * * *")
Data retention strategies:
 - Group: playbook_run_log, Age to Keep: 180 days, Raw Delete: True, Enabled: True, ID: 4, Description: [SOAR System Policy] Delete stale debug and error logs from historical playbook runs

In the sample output, a data retention strategy has been scheduled to run every hour, for the playbook_run_log group, setting the Age to Keep to a value of 180 days.

Add

Apply data retention strategies to a model by using the add subcommand.

 phenv configure_db_maintenance add --help

Guidelines for adding a data retention strategy:

If you do not specify an audit_category or a container label, the created strategy will apply to all audit categories or container labels, if the strategy is applicable to containers or audit categories.
You cannot create a generic container data retention strategy (one that is for all labels) that will delete items before a label-specific strategy.
You cannot create a generic audit data retention strategy (one that is for all audit categories) that will delete items before a category specific strategy


Argument	Required?	Description
`--help, -h`	Optional	Displays help for this command, then exits.
`--target-group <TARGET GROUP>` `-t <TARGET GROUP>`	Required	Name of the target group for the data retention strategy. audit automation_run container indicator playbook_run_log
`--container-label <CONTAINER LABEL>`	Optional	Specific label to target if this strategy targets the `container` group. You can omit this argument if you want the data retention strategy to apply to all labels. Because container labels are customizable, labels can vary by deployment. Use `phenv configure_db_maintenance add --help` and check the `--container-label` output for the most updated list for your deployment. Check to make sure you're applying your data retention strategy to the correct label for your deployment.
`--audit-category <AUDIT CATEGORY>`	Optional	Specific audit category to target if this strategy targets the `audit_category` group. You can omit this argument if you want the data retention strategy to apply to all audit categories. For a list of audit categories, see the Audit categories table. Audit categories may change from release to release. Your deployment may have more, fewer, or different categories than listed in the Audit categories table. Use `phenv configure_db_maintenance add --help` and check the `--audit-category` output for the most updated list for your deployment.
`--age-to-keep-time-unit <TIME UNIT>`	Required	The time unit for measuring `max-age-to-keep`. minutes hours days months years
`--max-age-to-keep <MAX AGE TO KEEP>`	Required	The maximum number of time units of database records to keep when applying the data retention strategy. You must set a value for `--age-to-keep-time-unit <TIME UNIT>`.
`--description <DESCRIPTION>`	Optional	A description of the data retention strategy.
`--disabled`	Optional	Disable the data retention strategy.

CAUTION: If you add a data retention strategy for a model that already has one, the new strategy replaces the existing strategy.

Edit

Modify data retention strategies by using the edit subcommand.

 phenv configure_db_maintenance edit --help


Argument	Required?	Description
`--help, -h`	Optional	Displays help for this command, then exits.
`--target-group <TARGET GROUP>` `-t <TARGET GROUP>`	Required	Name of the target group for the data retention strategy. audit automation_run container indicator playbook_run_log Use additional arguments when: your data retention strategy targets a specific label in the `container` group, you must also use the `--container-label` argument. If you want your data retention strategy to apply to all containers, do not use the `--container-label` argument. your data retention strategy targets specific audit items in the `audit` group, you must also use the `--audit-category` argument. If you want If you want your data retention strategy to apply to all audit items, do not use the `--audit-category` argument.
`--container-label <CONTAINER LABEL>`	Optional	Specific label to target if this strategy targets the `container` group. You can omit this argument if you want the data retention strategy to apply to all labels. Because container labels are customizable, labels can vary by deployment. Use `phenv configure_db_maintenance edit --help` and check the `--container-label` output for the most updated list for your deployment. Check to make sure you're applying your data retention strategy to the correct label for your deployment.
`--audit-category <AUDIT CATEGORY>`	Optional	Specific audit category to target if this strategy targets the `audit_category` group. You can omit this argument if you want the data retention strategy to apply to all audit categories. For a list of audit categories, see the Audit categories table. Audit categories may change from release to release. Your deployment may have more, fewer, or different categories than listed in the Audit categories table. Use `phenv configure_db_maintenance add --help` and check the `--audit-category` output for the most updated list for your deployment.
`--delete`	Optional	Permanently delete the data retention strategy.
`--enable`	Optional	Set the selected data retention strategy to active.
`--disable`	Optional	Deactivate the data retention strategy.

Schedule

Set the schedule for data retention strategies by using the schedule subcommand.

phenv configure_db_maintenance schedule --help


Argument	Required?	Description
`--help, -h`	Optional	Displays help for this command, then exits.
`--cron-schedule <CRON_SCHEDULE>`	Required	How often to run the scheduled data retention strategy. Must be a cron schedule expression.

Audit categories

Audit categories table

This table lists the available audit categories for use with the add --audit-category and edit --audit-category commands. Audit categories may change from release to release. Your deployment may have more, fewer, or different categories than listed in the Audit categories table.


Audit categories
action_run	forwarder_group
aggregation_rule	indicator_cef_filter
app	license
app_run	note
approval	ph_user
artifact	playbook
asset	report
automation_proxy	report_run
automation_rule	role
automation_rule_playbook_association	role_permission
container	severity
container_attachment	splunk_idp_role_mapping
container_comment	splunk_role
container_pin	system_settings
container_status	tenant
custom_function	workbook_phase
decided_list	workbook_phase_template
es_pairing	workbook_task
evidence	workbook_task_template
feature_flag	workbook_template

Examples

Each example is for a specific task where you may want to list, add, edit, or schedule a data retention policy.

Example: Get a list of all data existing data retention strategies

Get a list of all data retention strategies by using the list subcommand.

phenv configure_db_maintenance list

Example: Add or update a strategy to delete indicator records

You may want to delete indicator records to improve database query performance, or to reduce the amount of disk space use by Splunk SOAR's PostgreSQL database.

Delete indicator records after three months.

phenv configure_db_maintenance add --target-group indicator --age-to-keep-time-unit months --max-age-to-keep 3

Example: Add or update a strategy to delete audit records

You may want to delete all audit records after your company's records keeping requirements no longer need them.

Add or update a data retention strategy to delete all audit records from any category after one year:

phenv configure_db_maintenance add --target-group audit --age-to-keep-time-unit years --max-age-to-keep 1

Example: Add or update a strategy to delete records for the app_run audit category

This example demonstrates deleting records from a single audit category after 90 days.

phenv configure_db_maintenance add --target-group audit --audit-category app_run --age-to-keep-time-unit days --max-age-to-keep 90

Example: Turn off an existing data retention strategy

This example demonstrates tuning off a single data retention strategy that applies to all containers.

phenv configure_db_maintenance edit --disable --target-group container

Example: Delete an existing data retention strategy

This example demonstrates how to delete a data retention strategy. This strategy applies to all containers with the events label.

phenv configure_db_maintenance edit --delete --target-group container --container-label events

Example: Change the schedule for a data retention strategy

This example changes the schedule to run data retention strategies to every two hours, on the hour.

phenv configure_db_maintenance schedule --cron-schedule "0 */2 * * *"

Documentation