Manage your PostgreSQL database with data retention strategies

Use data retention strategies to schedule and cleanup records in your Splunk SOAR (On-premises) PostgreSQL database.

Use configure_db_maintenance to set parameters for retaining or removing database entries in the Splunk SOAR (On-premises) PostgreSQL database.

  • A set of options is called a strategy.
  • Models are collected into groups.
  • Strategies are applied to groups.
Strategy
The set of configurable parameters that define when a record should be deleted, either automatically or when the db_maintenance tool runs.
Model
Any PostgreSQL database record or Django object is called a model. Models have characteristics that define what sort of information the model represents.
Group nameDescription
auditAudit logs. See

Enable and download audit trail logs in Splunk SOAR (On-premises).

automation_runAutomation run data including Playbook Run, Action Run, App Run, Custom Function Run, and playbook run statistics.
containerContainers and related records like Artifacts and Container Attachments. See

About Splunk SOAR (On-premises).

indicatorIndicators or Indicators of Compromise. See

About Splunk SOAR (On-premises).

playbook_run_logDebug and error logs from historical playbook runs.

To use the configure_db_maintenance.py tool, follow these steps:

  1. SSH to your Splunk SOAR (On-premises) instance. 
     SSH <username>@<phantom_hostname>
  2. Use the following tool to manage data deletion. 
     phenv configure_db_maintenance
  3. Add your desired argument to the data retention tool command line to list, add, edit or schedule data retention actions.
Note: On clustered systems, the configure_db_maintenance.py tool can be run from any node, but only the leader node runs the data retention strategy.

Data retention tool arguments

Append the --help argument to your tool to get information on the data retention tool arguments; 

  phenv configure_db_maintenance --help

Optional arguments

Use these optional arguments to manage your data retention strategy.

Argument Description
-h, --help Show this help message and exit.
-v {0,1,2,3}, --verbosity {0,1,2,3} Set the verbosity for the command's output.
  • 0 for minimal output
  • 1 for normal output
  • 2 for verbose output
  • 3 for very verbose output
--no-colorDo not use color in the command output.
--skip-checksDo not perform system checks.
SubcommandDescription
listList all existing data retention strategies and their schedules.
addAdd a new data retention strategy, or update a data retention strategy by updating all of that strategy's settings. This command has options. See the table for Add later in this topic.
edit Selectively update an existing data retention strategy. This command has options. See the table for Edit later in this topic.
scheduleAdd or update a scheduled job to service all data retention strategies. This command has options. See the table for Schedulelater in this topic.

configure_db_maintenance subcommands

Use configure_db_maintenance to list, add, edit or schedule data retention strategies.

List

Get a list of all data retention strategies by using the list subcommand.

phenv configure_db_maintenance list

Sample output

 Enabled data retention strategies will be serviced on the following cron schedule: Every hour UTC (cron string: "0 * * * *")
Data retention strategies:
 - Group: playbook_run_log, Age to Keep: 180 days, Raw Delete: True, Enabled: True, ID: 4, Description: [SOAR System Policy] Delete stale debug and error logs from historical playbook runs

In the sample output, a data retention strategy has been scheduled to run every hour, for the playbook_run_log group, setting the Age to Keep to a value of 180 days.

Add

Apply data retention strategies to a model by using the add subcommand.

phenv configure_db_maintenance add --help

Guidelines for adding a data retention strategy:

  • If you do not specify an audit_category or a container label, the created strategy will apply to all audit categories or container labels, if the strategy is applicable to containers or audit categories.
  • You cannot create a generic container data retention strategy (one that is for all labels) that will delete items before a label-specific strategy.
  • You cannot create a generic audit data retention strategy (one that is for all audit categories) that will delete items before a category specific strategy
ArgumentRequired?Description
--help, -hOptionalDisplays help for this command, then exits.
--target-group <TARGET GROUP> -t <TARGET GROUP>RequiredName of the target group for the data retention strategy.
  • audit
  • automation_run
  • container
  • indicator
  • playbook_run_log
--container-label <CONTAINER LABEL> OptionalSpecific label to target if this strategy targets the container group. You can omit this argument if you want the data retention strategy to apply to all labels.
CAUTION: Because container labels are customizable, labels can vary by deployment. Use phenv configure_db_maintenance add --help and check the --container-label output for the most updated list for your deployment. Check to make sure you're applying your data retention strategy to the correct label for your deployment.
--audit-category <AUDIT CATEGORY>OptionalSpecific audit category to target if this strategy targets the audit_category group. You can omit this argument if you want the data retention strategy to apply to all audit categories.
Audit categories
action_runforwarder_group
aggregation_ruleindicator_cef_filter
applicense
app_runnote
approvalph_user
artifactplaybook
assetreport
automation_proxyreport_run
automation_rulerole
automation_rule_playbook_associationrole_permission
containerseverity
container_attachmentsplunk_idp_role_mapping
container_commentsplunk_role
container_pinsystem_settings
container_statustenant
custom_functionworkbook_phase
decided_listworkbook_phase_template
es_pairingworkbook_task
evidenceworkbook_task_template
feature_flagworkbook_template
CAUTION: Because container labels are customizable, labels can vary by deployment. Use phenv configure_db_maintenance add --help and check the --audit-category output for the most updated list for your deployment. Check to make sure you're applying your data retention strategy to the correct label for your deployment.
--age-to-keep-time-unit <TIME UNIT>RequiredThe time unit for measuring max-age-to-keep.
  • minutes
  • hours
  • days
  • months
  • years
--max-age-to-keep <MAX AGE TO KEEP>RequiredThe maximum number of time units of database records to keep when applying the data retention strategy. You must set a value for --age-to-keep-time-unit <TIME UNIT>.
--description <DESCRIPTION>OptionalA description of the data retention strategy.
--disabledOptionalDisable the data retention strategy.
CAUTION: If you add a data retention strategy for a model that already has one, the new strategy replaces the existing strategy.

Edit

Modify data retention strategies by using the edit subcommand.

phenv configure_db_maintenance edit --help
ArgumentRequired?Description
--help, -hOptionalDisplays help for this command, then exits.
--target-group <TARGET GROUP> -t <TARGET GROUP>RequiredName of the target group for the data retention strategy.
  • audit
  • automation_run
  • container
  • indicator
  • playbook_run_log

Use additional arguments when:

  • your data retention strategy targets a specific label in the container group, you must also use the --container-label argument. If you want your data retention strategy to apply to all containers, do not use the --container-label argument.
  • your data retention strategy targets specific audit items in the audit group, you must also use the --audit-category argument. If you want If you want your data retention strategy to apply to all audit items, do not use the --audit-category argument.
--container-label <CONTAINER LABEL>OptionalSpecific label to target if this strategy targets the container group. You can omit this argument if you want the data retention strategy to apply to all labels.
CAUTION: Because container labels are customizable, labels can vary by deployment. Use phenv configure_db_maintenance edit --help and check the --container-label output for the most updated list for your deployment. Check to make sure you're applying your data retention strategy to the correct label for your deployment.
--audit-category <AUDIT CATEGORY>Optional Specific audit category to target if this strategy targets the audit_category group. You can omit this argument if you want the data retention strategy to apply to all audit categories.
Audit categories
action_runforwarder_group
aggregation_ruleindicator_cef_filter
applicense
app_runnote
approvalph_user
artifactplaybook
assetreport
automation_proxyreport_run
automation_rulerole
automation_rule_playbook_associationrole_permission
containerseverity
container_attachmentsplunk_idp_role_mapping
container_commentsplunk_role
container_pinsystem_settings
container_statustenant
custom_functionworkbook_phase
decided_listworkbook_phase_template
es_pairingworkbook_task
evidenceworkbook_task_template
feature_flagworkbook_template
CAUTION: Audit categories may change from release to release. Your deployment may have more, fewer, or different categories than listed here. Use phenv configure_db_maintenance edit --help and check the --audit-category output for the most updated list for your deployment.
--deleteOptionalPermanently delete the data retention strategy.
--enableOptionalSet the selected data retention strategy to active.
--disableOptionalDeactivate the data retention strategy.

Schedule

Set the schedule for data retention strategies by using the schedule subcommand.

phenv configure_db_maintenance schedule --help
ArgumentRequired?Description
--help, -hOptionalDisplays help for this command, then exits.
--cron-schedule <CRON_SCHEDULE>RequiredHow often to run the scheduled data retention strategy. Must be a cron schedule expression.

Examples

Each example is for a specific task where you may want to list, add, edit, or schedule a data retention policy.

Example: Get a list of all data existing data retention strategies

Get a list of all data retention strategies by using the list subcommand.

phenv configure_db_maintenance list

Example: Add or update a strategy to delete indicator records

You may want to delete indicator records to improve database query performance, or to reduce the amount of disk space use by Splunk SOAR's PostgreSQL database.

Delete indicator records after three months.

phenv configure_db_maintenance add --target-group indicator --age-to-keep-time-unit months --max-age-to-keep 3

Example: Add or update a strategy to delete audit records

You may want to delete all audit records after your company's records keeping requirements no longer need them.

Add or update a data retention strategy to delete all audit records from any category after one year:

phenv configure_db_maintenance add --target-group audit --age-to-keep-time-unit years --max-age-to-keep 1

Example: Add or update a strategy to delete records for the app_run audit category

This example demonstrates deleting records from a single audit category after 90 days.

phenv configure_db_maintenance add --target-group audit --audit-category app_run --age-to-keep-time-unit days --max-age-to-keep 90

Example: Turn off an existing data retention strategy

This example demonstrates tuning off a single data retention strategy that applies to all containers.

phenv configure_db_maintenance edit --disable --target-group container

Example: Delete an existing data retention strategy

This example demonstrates how to delete a data retention strategy. This strategy applies to all containers with the events label.

phenv configure_db_maintenance edit --delete --target-group container --container-label events

Example: Change the schedule for a data retention strategy

This example changes the schedule to run data retention strategies to every two hours, on the hour.

phenv configure_db_maintenance schedule --cron-schedule "0 */2 * * *"