Enable private connectivity

1. Confirm eligibility

To confirm your eligibility for private connectivity, send an HTTP GET request to the following endpoint:

GET /{stack}/adminconfig/v2/private-connectivity/eligibility

For example, to check if TestStack is eligible, send the following request:

curl --location --request GET 'https://admin.splunk.com/TestStack/adminconfig/v2/private-connectivity/eligibility' \
--header 'Authorization: Bearer abcdefgh123456....'

The request returns a success or failure result based on whether the Splunk Cloud Platform deployment meets the required eligibility criteria.

If your Splunk Cloud Platform deployment is eligible, the request returns:

{
"eligible": true
}

If your Splunk Cloud Platform deployment is not eligible, the request returns:

{
"eligible": false
}

2. Enable private connectivity

Once you confirm that your Splunk Cloud Platform deployment is eligible for private connectivity, send an HTTP POST request to the following endpoint:

POST /{stack}/adminconfig/v2/private-connectivity/endpoints

This request requires you to specify an AWS account ID, Azure subscription ID, or Google Project ID and a feature that you want to associate with the private connectivity endpoint. For example, to enable private connectivity for both ingest and search for TestStack with your cloud provider-specific ID:

### AWS
curl --location --request POST 'https://admin.splunk.com/TestStack/adminconfig/v2/private-connectivity/endpoints'  \
-H "Content-Type: application/json" --header 'Authorization: Bearer abcdefgh123456...' \
--data-raw '{
"customerAccountIds" : ["112233445566"],
"feature" : ["ingest", "search"]
}'

### Azure
curl --location --request POST 'https://admin.splunk.com/TestStack/adminconfig/v2/private-connectivity/endpoints'  \
-H "Content-Type: application/json" --header 'Authorization: Bearer abcdefgh123456...' \
--data-raw '{
"customerAccountIds" : ["9068c06d-21f8-bcc3-4e68-c569b514b4b6"],
"feature" : ["ingest", "search"]
}'

### GCP
curl --location --request POST 'https://admin.splunk.com/TestStack/adminconfig/v2/private-connectivity/endpoints'  \
-H "Content-Type: application/json" --header 'Authorization: Bearer abcdefgh123456...' \
--data-raw '{
"customerAccountIds" : ["my-gcp-project-id"],
"feature" : ["ingest", "search"]
}'

The API creates a new endpoint and adds cloud IDs to the endpoints. The request returns one or more cloud IDs that were added to the endpoints:

{
"customerAccountIds" : ["112233445566"],
"feature" : ["ingest", "search"]
}

2.a Retrieve endpoint name

To confirm that private connectivity has been enabled and to retrieve the endpoint details, send an HTTP GET request to the following endpoint:

GET /{stack}/adminconfig/v2/private-connectivity/endpoints

For example, to retrieve the VPC endpoint ID for TestStack for all features:

curl --location --request GET 'https://admin.splunk.com/TestStack/adminconfig/v2/private-connectivity/endpoints?feature=all' \
--header 'Authorization: Bearer abcdefgh123456...'

When private connectivity for ingest and search are successfully enabled, the request returns the cloud account IDs, the name of the private virtual network endpoint, (VPC endpoint in AWS, or Resource ID in Azure S2S forwarder) or Resource ID and Target Sub Resource (Azure HEC and Search), a message confirming enablement, and an available status.

AWS

On AWS, the request returns the AWS account IDs and the name of the VPC endpoint (or endpoints depending on features enabled). For example:

### AWS
{
    "endpoints":[
        {
        "customerAccountIds": [
            "112233445566",
            "778899101011"
        ],
        "endpoint": "com.amazonaws.vpce.us-east-1.vpce-svc-00f4e0098fce35fb9",
        "endpointV6": "com.amazonaws.vpce.us-east-1.vpce-svc-0868200e62b466d7d",
        "feature":"ingest",
        "message": "Private Connectivity for ingest is enabled. Please refer to 
        https://docs.splunk.com/Documentation/SplunkCloud/latest/Security/Privateconnectivity 
        for more information.",
        "status": "available"
        },
        {
        "customerAccountIds": [
            "112233445566",
            "778899101011"
        ],
        "endpoint": "com.amazonaws.vpce.us-east-1.vpce-svc-0d0f607d0a67e50f9",
        "endpointV6": "com.amazonaws.vpce.us-east-1.vpce-svc-05c2509de92b02472", 
        "feature":"search",
        "message": "Private Connectivity for search is enabled. Please refer to 
        https://docs.splunk.com/Documentation/SplunkCloud/latest/Security/Privateconnectivity 
        for more information.",
        "privateSearch-DNSRecords":[
            "sh1.pvt.TestStack.splunkcloud.com",
            "shc1.pvt.TestStack.splunkcloud.com",
            "pvt.TestStack.splunkcloud.com"
        ],
        "status": "available"
}]

Azure

On Azure, the request returns the Azure subscription IDs. The Azure S2S forwarder returns the Resource ID and Azure HEC and Search returns the Resource ID as well as a Target Sub Resource. For example:

### Azure
{
    "endpoints":[
        {
        "customerAccountIds": [
            "7ea26c8a-8ffb-42da-bb7d-7f3992ba3aba",
            "c9c82e13-25a5-4bc4-a4ac-adf8ee343100"
        ],
        "feature":"ingest",
        "message": "Private Connectivity for ingest is enabled. Please refer to 
https://docs.splunk.com/Documentation/SplunkCloud/latest/Security/Privateconnectivity 
for more information.",
        "resourceId":
"/subscriptions/ab2b15d7-1553-4d97-8e6f-df7859dcad17/resourceGroups/TestStack-ingest-lb-rg/providers/Microsoft.Network/privateLinkServices/TestStack-s2s-pl-svc",
        "status": "available"
        },
        {
        "customerAccountIds": [
            "7ea26c8a-8ffb-42da-bb7d-7f3992ba3aba",
            "c9c82e13-25a5-4bc4-a4ac-adf8ee343100"
        ],
        "feature":"ingest",
        "message": "Private Connectivity for ingest is enabled. Please refer to 
https://docs.splunk.com/Documentation/SplunkCloud/latest/Security/Privateconnectivity 
for more information.",
        "resourceId":
"/subscriptions/ab2b15d7-1553-4d97-8e6f-df7859dcad17/resourceGroups/TestStack-hec-appgw-rg/providers/Microsoft.Network/applicationGateways/TestStack-hec-appgw",
        "status": "available",
        "targetSubResource": "TestStack-hec-feip"
        },
        {
        "customerAccountIds": [
            "7ea26c8a-8ffb-42da-bb7d-7f3992ba3aba",
            "c9c82e13-25a5-4bc4-a4ac-adf8ee343100"
       ],
        "feature":"search",
        "message": "Private Connectivity for search is enabled. Please refer to 
https://docs.splunk.com/Documentation/SplunkCloud/latest/Security/Privateconnectivity 
for more information.",
        "privateSearch-DNSRecords": [
            "pvt.TestStack.splunkcloud.com",
            "sh1.pvt.TestStack.splunkcloud.com"
        ],
	"resourceId": 
"/subscriptions/ab2b15d7-1553-4d97-8e6f-df7859dcad17/resourceGroups/TestStack-search-appgw-rg/providers/Microsoft.Network/applicationGateways/TestStack-search-appgw",        
        "status": "available",
        "targetSubResource": "TestStack-search-feip"
	}
]

GCP

On GCP, the request returns the Google Project IDs, the VPC Service Endpoint ID, and the VPC Service Name for each feature. The privateSearch-DNSRecords returned for the search feature are used to configure DNS routing within your private virtual network. GCP leverages disparate endpoints for search web and search API traffic, ensure the appropriate DNS records resolve to the correct endpoint.

### GCP
{
    "endpoints": [
        {
            "customerAccountIds": ["conductorless-3-01eb"],
            "feature": "search",
            "message": "Private Connectivity for search is enabled. Please refer to https://docs.splunk.com/Documentation/SplunkCloud/latest/Security/Privateconnectivity for more information.",
            "privateSearch-DNSRecords": [
                "api-psc.pvt.restful-rabbit-yn3.splunkworks.lol",
                "sh1-api-psc.pvt.restful-rabbit-yn3.splunkworks.lol",
                "shc1-api-psc.pvt.restful-rabbit-yn3.splunkworks.lol"
            ],
            "resourceId": "projects/restful-rabbit-yn3-53d0/regions/us-central1/serviceAttachments/restful-rabbit-yn3-search-api-psc-attachment",
            "status": "available"
        },
        {
            "customerAccountIds": ["conductorless-3-01eb"],
            "feature": "search",
            "message": "Private Connectivity for search is enabled. Please refer to https://docs.splunk.com/Documentation/SplunkCloud/latest/Security/Privateconnectivity for more information.",
            "privateSearch-DNSRecords": [
                "pvt.restful-rabbit-yn3.splunkworks.lol",
                "sh1.pvt.restful-rabbit-yn3.splunkworks.lol",
                "shc1.pvt.restful-rabbit-yn3.splunkworks.lol"
            ],
            "resourceId": "projects/restful-rabbit-yn3-53d0/regions/us-central1/serviceAttachments/restful-rabbit-yn3-search-web-psc-attachment",
            "status": "available"
        }
    ]
}

If private connectivity has not been enabled, the request output shows the status of the request and a reason for that status. For example:

Private connectivity for ingest is not enabled:

{
    "endpoints":[
        {
        "feature":"ingest",
        "status": "unavailable",
        "reason": "uninitialized"
        }
    ]
}

Private connectivity for ingest initialization request is still in progress:

{
    "endpoints":[
        {
        "feature":"ingest",
        "status": "unavailable",
        "reason": "initialization is in progress"
        }
    ]
}

Private connectivity for ingest request failed:

{
    "endpoints":[
        {
        "feature":"ingest",
        "status": "unavailable",
        "reason": "initialization failed"
        }
    ]
}

3. (For Data Ingest only) Download universal forwarder credentials for private connectivity

Once you have enabled Splunk Cloud Platform private connectivity, follow the instructions in Deploy the universal forwarder to configure your universal forwarders to connect with your cloud provider's private link technologies. Download the private connectivity package for each universal forwarder that you want to use to send data to private endpoints.

To confirm your forwarder is configured for private connectivity, check the server value in outputs.conf.

• Non-private connectivity: inputs1.<stack-name>.splunkcloud.com, inputs2.<stack-name>.splunkcloud.com, ...
• Private connectivity: inputs1.pvt.<stack-name>.splunkcloud.com, inputs2.pvt.<stack-name>.splunkcloud.com, ...

4. Create private virtual network endpoint and enable private link

Now that you have configured your universal forwarders for your cloud provider's private link technology, follow the instructions from your cloud provider to create endpoints for the private connectivity features that you enabled.

• Access an inspection system using a Gateway Load Balancer endpoint - AWS PrivateLink

• Quickstart: Create a private endpoint by using the Azure portal

• Google Private Service Connect documentation

For ingest

AWS

• In PrivateLink Ready partner services, add the VPC endpoint name provided by Splunk in Step 2.a under "feature : Ingest" in the Service Name field.
• Set IP address type appropriately for the endpoint service. For example, IPv4 for 'endpoint' and IPv6 for 'endpointv6' (if your stack is dual-stack enabled).
• For security group, select a security group that allows ports 443 and 9997 as inbound, or create a new security group and then return to this page and refresh the list.
• In endpoint service, under the details section, take note of the first DNS record entry with the vpce-xxx.vpce-svc-xxx prefix (in other words, the first entry that does not include the availability zones). You use this to configure your private DNS zone. For example: com.amazonaws.vpce.us-east-1.vpce-svc-10022f155a5dd8b21

Azure

• In Private Link Center | Private Endpoints, select Connect to an Azure resource by resource ID or alias radial, then add the resource ID (S2S) or resource ID and target sub resource (HEC) provided by Splunk in Step 2.a under "feature : Ingest" in the fields.
• For security groups, allow port 443 and 9997 as outbound any subnets accessing the endpoint. In Private Link Center | Private Endpoints, in the Private IP column, take note of private IP address that has been assigned to the given endpoint. You use this to configure your private DNS zone and entries. For example: 10.1.2.3

GCP

• In Google Cloud Console, navigate to Network services > Private Service Connect.
• Create a PSC endpoint using the service attachment ID provided by Splunk in Step 2.a under "feature : Ingest."
• In your Google Cloud Firewall, create rules allowing egress traffic on ports 443 and 9997 from resources that will send data via PSC.
• Note the private IP address assigned to the PSC endpoint. You use this to configure your private DNS zone.

For search

Set up a new endpoint service to receive search results.

AWS

• In Other endpoint services, add the VPC endpoint name provided by Splunk in Step 2.a under "feature : Search" in the Service Name field.
• Set IP address type to the appropriate IP version in accordance with the endpoint service you are connecting to. For example, IPv4 for for 'endpoint' and IPv6 for 'endpointv6' (if your stack is dual-stack enabled)IPv4..
• For security groups, allow port 80, 443 and 8089 as inbound.
• In endpoint service, under the details section, take note of the first DNS record entry with the vpce-xxx.vpce-svc-xxx prefix (in other words, the first entry that does not include the availability zones). You use this to configure your private DNS zone. For example: vpce-svc-10036e133a3cc1f94.vpce-svc-0ed7f8a09d0fef69b.us-east-1.vpce.amazonaws.com

Azure

• In Private Link Center | Private Endpoints, select Connect to an Azure resource by resource ID or alias radial, then add the resource ID and target sub resource (search) provided by Splunk in Step 2.a under "feature : Ingest" in the fields.
• For security groups, allow 80, port 443 and 8089 as outbound any subnets accessing the endpoint. In Private Link Center | Private Endpoints, in the Private IP column, take note of private IP address that has been assigned to the given endpoint. You use this to configure your private DNS zone and entries. For example: 10.1.2.3

GCP

• In Google Cloud Console, navigate to Network services > Private Service Connect.
• Create a PSC endpoint using the service attachment ID provided by Splunk in Step 2.a under "feature : Search."
• In your Google Cloud Firewall, create rules allowing egress traffic on ports 443 and 8089 from resources that will access the search tier via PSC.
• Note the private IP address assigned to the PSC endpoint. You use this to configure your private DNS zone.

5. Configure DNS name

To resolve Splunk forwarder or search traffic to the Splunk Cloud Platform from your endpoint services, you must configure appropriate DNS mappings for each cloud provider account that you want to connect to your private link endpoints.

Create a private hosted zone with one of the following domain names:

• Splunk Cloud Platform deployments in standard regions: pvt.<stack-name>.splunkcloud.com

• Splunk Cloud Platform deployments in GovCloud regions: pvt.<stack-name>.splunkcloudgc.com

For ingest:

AWS

• Create a wildcard CNAME that points to your VPCE endpoint or create an A record type of ' * ' for S2S and point it to the private IPv4 addresses associated with the private link endpoint you created in the previous section (for example: 10.1.2.3).
• Create an additional A record type of ' http-inputs' for HEC and point it to the private IPv4 addresses associated with the private link endpoint you created in the previous section (for example: 10.1.2.3).

• *Only required if IPv6 feature is enabled on your stack)

  Create an AAAA record type of ' * ' for S2S and point it to the private IPv6 addresses associated with the private link endpoint you created in the previous section (for example: 2d2e:a3e2:2e9f:f94f:592a:1808:a575:ca5c).

• Create an additional AAAA record type of 'http-inputs' for HEC and point it to the private IPv6 addresses associated with the private link endpoint you created in the previous section (for example: 2d2e:a3e2:2e9f:f94f:592a:1808:a575:ca5c).
• *Caution: If you previously created a CNAME for IPv4 DNS resolution for either S2S or HEC, those records need to be updated to A records prior to creating AAAA records for IPv6, as RRSET conflicts might occur otherwise.

Azure

• Create a wildcard CNAME that points to your VPCE endpoint or an A record type for ' * ' and point it to the private IP address associated with the private link endpoint associated with the forwarder (S2S) resource (for example: 10.1.2.3).
• Create an additional A record type for 'http-inputs' and point it to the private IP address associated with the private link endpoint associated with the HEC resource (for example: 10.2.3.4).

GCP

• Create a wildcard A record ( * ) in your private DNS zone pvt.<stack-name>.splunkcloud.com pointing to the private IP address of the PSC S2S endpoint (for example: 10.1.2.3).
• Create an additional A record for http-inputs pointing to the private IP address of the PSC HEC endpoint (for example: 10.1.2.3).

For search:

AWS

• Make a note of all the DNS records associated with your search tier from Step 2a under the "Feature: Search" section.
• Create an A record type for each DNS record obtained in the previous step and point them to the private IPv4 addresses associated with the private link endpoint you created in the previous section (for example: 10.2.3.4).
• *Only required if IPv6 feature is enabled on your stack)
• Create an AAAA record type for each DNS record obtained in the previous step and point them to the private IPv6 addresses associated with the private link endpoint you created in the previous section (for example: 2c5c:e244:0864:3f0a:ea67:6ad1:ab0c:aae8).
• *Caution: If you previously created a CNAME for IPv4 DNS resolution for any search DNS records, those records need to be updated to A records prior to creating AAAA records for IPv6, as RRSET conflicts might occur otherwise.

Azure

• Make a note of all the DNS records associated with your search tier from Step 2a under the "Feature: Search" section.
• Create an A record type for each DNS record obtained in the previous step and private IP address associated with private link endpoint associated with the search resource (for example: 10.3.4.5).

GCP

• Note all DNS records associated with your search tier from Step 2.a under "Feature: Search."
• Create an A record for each DNS record in your private zone, pointing to the private IP address of the PSC search and API endpoints (for example: 10.2.3.4).

• There is a separate endpoint for search and API. You must create DNS entries for each.

Your Splunk Cloud Platform deployment is now configured for private connectivity.

(Optional) Add additional cloud account IDs

If you need to add additional cloud account IDs to the private link endpoint, send an HTTP PATCH request to the following endpoint:

PATCH /{stack}/adminconfig/v2/private-connectivity/endpoints

For example, to add an additional cloud account to the private link endpoint(s) for TestStack for feature ingest:

### AWS
curl --location --request PATCH 'https://admin.splunk.com/{stack}/adminconfig/v2/private-connectivity/endpoints' \
-H "Content-Type: application/json" --header 'Authorization: Bearer abcdefgh123456...' \
--data-raw '{
"customerAccountIds" : ["123456789101"],
"feature" : ["ingest", "search"]
}'

### Azure
curl --location --request PATCH 'https://admin.splunk.com/{stack}/adminconfig/v2/private-connectivity/endpoints' \
-H "Content-Type: application/json" --header 'Authorization: Bearer abcdefgh123456...' \
--data-raw '{
"customerAccountIds" : ["2d585d36-c8c6-ae66-a226-33460a6cf3e5"],
"feature" : ["ingest", "search"]
}'

### GCP
curl --location --request PATCH 'https://admin.splunk.com/{stack}/adminconfig/v2/private-connectivity/endpoints' \
-H "Content-Type: application/json" --header 'Authorization: Bearer abcdefgh123456...' \
--data-raw '{
"customerAccountIds" : ["another-gcp-project-id"],
"feature" : ["ingest", "search"]
}'

The request returns the cloud account ID(s) that were added to the endpoints:

### AWS
{
    "feature": [
        "ingest", "search"
    ],
    "patchedCustomerAccountIds": [
        "123456789101"
    ]
}

### Azure
{
    "feature": [
        "ingest", "search"
    ],
    "patchedCustomerAccountIds": [
        "2d585d36-c8c6-ae66-a226-33460a6cf3e5"
    ]
}

### GCP
{
    "feature": [
        "ingest", "search"
    ],
    "patchedCustomerAccountIds": [
        "another-gcp-project-id"
    ]
}

(Optional) SAML configuration

Private connectivity--for search or data ingest--creates a private route (using the cloud provider's private link technology) to search or ingest your data in addition to the publicly-accessible route. Stacks configured for search using private connectivity can use SAML in one of two ways: mixed-mode or private link-only.

Mixed-mode SAML

Private search-enabled stacks can use SAML in a bimodal mode. Because most SAML IdPs only support a single redirect, this configuration requires that both Splunk and the SAML IdP have been updated to redirect to private search tier DNS entries. Clients will then rely on DNS resolution to resolve either the internet-facing IPs or the VPC endpoint IPs as applicable, respectively. When mixed-mode is enabled, a Splunk-managed private zone will be created that redirects to the regular, non-private, zone and clients accessing the search tier using private link will rely on the customer-managed private zone resolving to their endpoint for private search.

Internet

DNS resolution flow

1. Client browses <search_tier>.<stack-name>.<domain>.
2. Search tier's SAML configuration sends auth to IdP.
3. Upon successful auth, IdP redirects to <search_tier>.pvt.<stack-name>.<domain>.
4. Client resolves <search_tier>.pvt.<stack-name>.<domain> in Splunk-managed private DNS zone, which are a CNAME(s) resolving to <search_tier>.<stack-name>.<domain>, the Splunk-managed primary DNS zone.
5. Client logs in with appropriate authorization using SAML over the Internet.
6. The client will see the URL change, which is expected, given the configuration. Despite the <search_tier>.pvt.<stack-name>.<domain> in the URL, traffic will traverse the Internet.

Private Link

DNS resolution flow

1. Client browses <search_tier>.<stack-name>.<domain>.
2. Search tier's SAML configuration sends auth to IdP. The VPC or direct connection must allow traffic to reach the SAML IdP.
3. Upon successful auth, IdP redirects to <search_tier>.pvt.<stack-name>.<domain>.
4. Client resolves address <search_tier>.pvt.<stack-name>.<domain> in customer-managed private DNS zone, which ultimately resolves to the VPC endpoint IP(s) from the newly created endpoint.
5. Client logs in with appropriate authorization using SAML over the private link.

AWS

This diagram illustrates PrivateLink Search using a SAML IdP to authenticate.

Azure

This diagram illustrates Azure Search using a SAML IdP to authenticate.

GCP

This diagram illustrates GCP Private Service Connect Search using a SAML IdP to authenticate.

PrivateLink-only / PSC-only SAML

To configure SAML for use in a stack where Private Search is enabled and mixed-mode is set to false, ensure that Splunk and the SAML IdP have been updated to redirect to the private search tier CNAME(s) created in the steps above. DNS must resolve appropriately to the newly created private endpoint for the SAML handshake to complete properly.

For GCP, ensure that Splunk and your SAML IdP have been updated to redirect to the private search tier DNS entries created in Step 5 above (<search_tier>.pvt.<stack-name>.splunkcloud.com). DNS must resolve to the PSC endpoint IP for the SAML handshake to complete.

Data Ingest

Use the following best practices if you send data to your Splunk Cloud Platform with:

HEC or serverless endpoints

• AWS Lambda function, Azure Function App, or Google Cloud Function must be bound to your private virtual network.
• Use the suggested HEC / serverless endpoint URLs:
  • Private endpoint: http-inputs.pvt.<stackname>.splunkcloud.com
  • Public endpoint: http-inputs.<stackname>.splunkcloud.com
• Traffic must be allowed through the security group or firewall associated with your serverless endpoint.

AWS Direct Connect, Azure ExpressRoute, Google Cloud VPN/Router or similar technologies

For AWS, consider the following:

• Set up an additional DNS configuration to send data from on-premise targets via AWS VPC using AWS PrivateLink.
• Configure AWS Route 53 or similar inbound resolver to resolve inputs hostnames for your on-premise forwarder.

Routing traffic from different cloud regions

AWS

The interface VPC endpoint (PrivateLink) can be created in the VPC that is located in the same region as your Splunk Cloud Platform deployment or another Splunk-supported region.

Splunk-supported regions include:

• ap-northeast-1
• ap-northeast-2
• ap-south-1
• ap-southeast-1
• ap-southeast-2
• ap-southeast-3
• ca-central-1
• eu-central-1
• eu-north-1
• eu-south-1 (Milan)
• eu-west-1
• eu-west-2
• eu-west-3
• sa-east-1 (Brazil)
• us-east-1
• us-east-2
• us-west-2

Azure

The Azure cloud provider makes use of virtual networking, such that private endpoints are not required to be created within the same region as the Splunk Cloud Platform stack. For connectivity from different regions, simply create a new Private Link endpoint attached to a private virtual network in the desired region.

GCP

Cross-region private connectivity is supported for GCP and is available across all Google Cloud regions. To send traffic from workloads in a different region:

• Create the PSC connection in a subnet that is in the same region as your Splunk Cloud Platform deployment. A dedicated subnet is not required.
• Enable Global Access on your PSC connections where cross-region access is required. This setting is available when creating or editing the PSC connection in the Google Cloud Console.
• In your Google Cloud Firewall, allow ingress traffic to the PSC subnet on TCP ports 443 and 9997 (ingest) or 443 and 8089 (search) from the CIDR ranges of your cross-region subnets.