Known issues - Splunk Documentation (2024)

Workaround:
An enhancement was introduced to the search process in Splunk Enterprise version 9.1.1 that optimizes searches by using the peer's IP address instead of querying DNS for the target peers. If an http_proxy is specified in the server.conf file, the enhancement causes the originating peer's IP address to fail to resolve. As a result, the IP address of the proxy, instead of the originating peer, is logged and utilized in the peers.csv file during search operations, causing the following error message to display in Splunk Web: "Received error from proxy server".

To disable the new DNS query optimization and eliminate the error, add the following setting to the distsearch.conf file:

[distributedSearch]useIPAddrAsHost=false

Workaround:
Apps may experience this issue if they: implement a custom search command using the Splunk Enterprise SDK for Python between versions 1.6.5 and 1.6.13; are executed by Splunk Enterprise or Splunk Cloud using Python 3; and are sent events with multi-byte characters.

App developers whose apps implement a custom search command using a version of the Splunk Enterprise SDK for Python must update to version 1.6.14 or higher and release new versions of their apps.

Splunk Enterprise and Splunk Cloud administrators who are using apps impacted by this issue must update to app versions that use the Splunk Enterprise SDK for Python version 1.6.14 or higher. If this is not possible, administrators are encouraged to either: allow these apps to be executed using Python 2; or cease usage of impacted apps until updated versions are available.

Workaround:
Update dpkg to version 1.17.6 or later.

Workaround:
Create a $SPLUNK_HOME/etc/system/local/user-seed.conf and restart Splunk

[user_info]
PASSWORD = <yourpassword>

Workaround:
Two possible approaches:

1. Remove the option grantableRoles = admin from authorize.conf - this is not permanent workaround and will need to be done every time admin role is modified.

2. Add any capabilities that the other user roles have to the "admin" role.

Workaround:
Set limits.conf back to default, by removing any override of max_searches_per_process.

For example:

[search]max_searches_per_process=1

to

[search]

Workaround:
Create a $SPLUNK_HOME/etc/system/local/user-seed.conf and restart Splunk

[user_info]
PASSWORD = <yourpassword>

Workaround:
Through the command line, you can open notepad and modify the password file to regain 'Admin' status.

Workaround:
When using a search with only the timechart command in a search:

| timechart count

Splunk instance will crash with "Crashing thread: searchOrchestrator". Currently there is no workaround other than not using this search string

Workaround:
The tstats command currently doesn't filter events with CIDR match on fields that contain IPv6 addresses. Running tstats searches containing IPv6 addresses might result in the following error indicating that the addresses are treated as non-exact queries:

Error in 'TsidxStats': WHERE clause is not an exact query

Workaround:
An enhancement was introduced to the search process in Splunk Enterprise version 9.1.1 that optimizes searches by using the peer's IP address instead of querying DNS for the target peers. If an http_proxy is specified in the server.conf file, the enhancement causes the originating peer's IP address to fail to resolve. As a result, the IP address of the proxy, instead of the originating peer, is logged and utilized in the peers.csv file during search operations, causing the following error message to display in Splunk Web: "Received error from proxy server".

To disable the new DNS query optimization and eliminate the error, add the following setting to the distsearch.conf file:

[distributedSearch]useIPAddrAsHost=false

Workaround:
Perform field extractions by modifying your searches using other commands, such as the rex command or eval command.

Workaround:
Ad hoc searches searches that use the earliest time modifier with a relative time offset should also include 'latest=now' in order to avoid time range inaccuracies. For example, if you want to get all events from the last 10 seconds starting at 01:00:10, the following search returns all events that occur between the time of 01:00:00 and 01:00:10, as expected.

index=main earliest=-10s latest=now

Running the same search without including 'latest=now' might produce unpredictable results or impact performance in certain scenarios when the search head is overloaded with ad hoc searches. See Specify earliest relative time offset and latest time in ad hoc searches in the Splunk platform Search Manual.

Workaround:
The setting precalculate_required_fields_for_alerts=0 can be set on saved searches that have no other alert actions attached aside from the "Run A Script" action, to quash the error. For saved searches that have multiple alert action attached, this may not be safe as it will disable back propagation of required fields for all alert actions, which might result in the parent search extracting more fields than required, which could negatively impact performance for that search.

Workaround:
In Splunk Enterprise 8.1.7, 8.2.4, and higher change the job_default_auto_cancel setting in $SPLUNK_HOME/etc/system/local/web.conf from the default value of 30 to 62.

Details
This issue is caused by power saving settings in recent browser versions, where Javascript timers may be throttled. The user typically sees the following message in the search window on foreground searches:

DAG Execution Exception: Search has been cancelled
Search auto-canceled
The search job has failed due to an error. You may be able to view the job in the Job Inspector

Workaround:
none

Workaround:
Set limits.conf back to default, by removing any override of max_searches_per_process.

For example:

[search]max_searches_per_process=1

to

[search]

Workaround:
Apps may experience this issue if they: implement a custom search command using the Splunk Enterprise SDK for Python between versions 1.6.5 and 1.6.13; are executed by Splunk Enterprise or Splunk Cloud using Python 3; and are sent events with multi-byte characters.

App developers whose apps implement a custom search command using a version of the Splunk Enterprise SDK for Python must update to version 1.6.14 or higher and release new versions of their apps.

Splunk Enterprise and Splunk Cloud administrators who are using apps impacted by this issue must update to app versions that use the Splunk Enterprise SDK for Python version 1.6.14 or higher. If this is not possible, administrators are encouraged to either: allow these apps to be executed using Python 2; or cease usage of impacted apps until updated versions are available.

Workaround:
Dedup values in search before, for example:

instead of

index="field_test" [search index="field_test" globalCallID_callId=1234* | fields globalCallID_callId]

add a stats or dedup in the subsearch:

index="field_test" [search index="field_test" globalCallID_callId=123* | stats values(globalCallID_callId) AS globalCallID_callId | mvexpand globalCallID_callId ]

If that list is still large and you're seeing the slowdown, consider moving the filtering to a | where after the initial search, for example:

index="field_test" globalCallID_callId=* | where [search index="field_test" globalCallID_callId=123* | stats values(globalCallID_callId) AS globalCallID_callId | mvexpand globalCallID_callId ]

Workaround:
- Increase globallimit to the value of "unique values" number mentioned in the warning message:

"The split by field <field> has a large number of unique values <number>. Chart column set will be trimmed to 10. Use globallimit argument to control column count."

- Use very high globallimit in geostats and post process after if needed

- Don't use BY in geostats

- Use lower cardinality BY and/or higher globallimit in geostats

Workaround:
One possible workaround is to use a more efficient query. For example, use "| tstats count where index=main by splunk_server" instead of "index=main | stats count by splunk_server".

Another workaround is to change the max_workers_searchparser setting to a value lower than its default.

Use this workaround if you are using your Splunk Enterprise federated search head (FSH) instance only for running federated searches. This workaround might affect non-federated searches.

On the Splunk Enterprise FSH, follow these steps:

1. Create limits.conf in a local/ folder.
2. Set the max_workers_searchparser setting to a number lower than its default (1 or 2). For more information about this setting see the Admin Manual.
3. Test which setting value provides a better performance.

Workaround:
Define the lookup on both federated search head and remote search head.

Workaround:
Define the same lookup on the remote search head too, so the remote search head will not error out early and return 0 results.

Workaround:
Use REST API to create the federated saved search instead:

curl -k -u <username>:<password> -X POST https://localhost:8089/servicesNS/nobody/search/data/federated/index -d name=federated:index_kathy -d federated.dataset='savedsearch:ss with space' -d federated.provider=remote_deployment_1
See [[Documentation:SplunkCloud:latest:RESTREF:RESTfederated|Federated search endpoint descriptions]] in the REST API Reference Manual.

Workaround:
Use Verbose and Smart mode specifically for searches with transforming commands like stats, chart, and timechart, and then review the results in the Statistics tab. To review event counts, run non-transforming searches in Fast mode.

Search-time field extraction usually requires searches without transforming commands that run in either Verbose or Smart mode. When you run searches in Fast mode, you can ensure that search-time field extraction takes place for federated searches by appending | fields * to the ends of your searches.

Workaround:
The workaround is to change the Data Model definition to reorder the root search objects such that the root search object that can be accelerated is the very first one in the list.

For instance, for the provided `test_internal_audit_logs.json`, edit the JSON file on disk and move `failed_searches` dataset before `fully_completed_searches`.

Workaround:
For a data model that is experiencing high memory usage, perform the following steps:

1. On your Splunk platform deployment, in Splunk Web, select Settings and then Data Models.
2. Select Edit for the data model that is experiencing high memory usage, and then select Edit Acceleration.
3. Open Advanced Settings.
4. Disable Automatic Rebuilds.

Furthermore, applying index constraints to restrict the list of indexes searched for building a given DMA summary and applying tags allowlisting would help curtail the memory usage.

Workaround:
Change object layers by manually editing the order of elements in the layout structure in the dashboard definition.

Workaround:
Resize the viewport or current monitor resolution. If the issue persists, try changing the dimensions of the dashboard.

Workaround:
See Troubleshoot referenced real-time searches for workaround details.

Workaround:
none

Workaround:
There is no workaround. After upgrading to Splunk Enterprise 8.x, the splunkd process checks and modifies the local/distsearch.conf on each service start. The process will:

• Remove any settings that define default values already set in the /default/distsearch.conf file.
• Removes comments preceded by a hash.
• Reorders the KV pairs alphanumerically within a stanza.
• Reorders stanzas within the file.

Workaround:
The workaround is to change the Data Model definition to reorder the root search objects such that the root search object that can be accelerated is the very first one in the list.

For instance, for the provided `test_internal_audit_logs.json`, edit the JSON file on disk and move `failed_searches` dataset before `fully_completed_searches`.

Workaround:
For a data model that is experiencing high memory usage, perform the following steps:

1. On your Splunk platform deployment, in Splunk Web, select Settings and then Data Models.
2. Select Edit for the data model that is experiencing high memory usage, and then select Edit Acceleration.
3. Open Advanced Settings.
4. Disable Automatic Rebuilds.

Furthermore, applying index constraints to restrict the list of indexes searched for building a given DMA summary and applying tags allowlisting would help curtail the memory usage.

Workaround:
when only one indexer in a cluster indexed data (has all the searchable copies), rebalance once before adding the new indexer, and then rebalance a second time

Workaround:
In Splunk Enterprise 8.1.7, 8.2.4, and higher change the job_default_auto_cancel setting in $SPLUNK_HOME/etc/system/local/web.conf from the default value of 30 to 62.

Details
This issue is caused by power saving settings in recent browser versions, where Javascript timers may be throttled. The user typically sees the following message in the search window on foreground searches:

DAG Execution Exception: Search has been cancelled
Search auto-canceled
The search job has failed due to an error. You may be able to view the job in the Job Inspector

Workaround:
Either proactively monitor and restart Splunk or kill splunkd process "instrument-resource-usage".

Alternatively, disable introspection altogether. In the introspection_generator_addon app

add the [introspection:generator:resource_usage] stanza in %SPLUNK_HOME%\etc\apps\introspection_generator_addon\local\server.confas follow: [introspection:generator:resource_usage]disabled = trueacquireExtra_i_data = false

Workaround:
If you are using addcoltotals to generate a totals data row, renaming the _time field can cause PDF generation issues.

Remove the label and labelfield or change the label to a number to generate the PDF as expected.

Workaround:
There is no workaround. After upgrading to Splunk Enterprise 8.x, the splunkd process checks and modifies the local/distsearch.conf on each service start. The process will:

• Remove any settings that define default values already set in the /default/distsearch.conf file.
• Removes comments preceded by a hash.
• Reorders the KV pairs alphanumerically within a stanza.
• Reorders stanzas within the file.

Workaround:
A timeout or slow performance of the license management page is caused by a build-up of historical license warning messages, which are processed every time the page is accessed. Can be verified by running this search on the License Manager:

| rest splunk_server=local /services/licenser/messages

If a high value is returned for that end point, you are likely affected. Log a support ticket with Splunk to obtain a license reset key, and apply the key to clear out any historical license warning messages. After the reset license is applied, the license management pages should load normally.

Workaround:
Navigate to a visible app, such as the search and reporting app, and access the Splunk settings pages from that app context.

Workaround:
When setting up maximum daily volume for this pool, choose 'The license maximum' option.

Workaround:
Disable Splunk Assist fully on those instances, see "Turn off Splunk Assist" in the Splunk Documentation for the procedure: https://docs.splunk.com/Documentation/Splunk/9.1.1/DMC/ActivateAssist#Turn_off_Splunk_Assist

Modular inputs can be disabled individually in $SPLUNK_HOME/etc/apps/splunk_assist/local/inputs.conf

Workaround:
Queued job displays using job endpoint with SID:

| rest /services/search/jobs/1668102339.174_23558BC9-6A39-4F4A-9FD2-968C358489B7 splunk_server=local

Workaround:
To prevent this issue: When you delete a workload pool, please make sure that you delete any disabled workload rules that are associated with that workload pool.

To resolve the issue if you encounter this: Disable or delete the workload rule that is associated with a workload pool that does not exist anymore.

Workaround:
In the volumes using gcp-sse-kms encryption mode, specify "remote.gs.upload_chunk_size = 0" to disable parallel upload.

Workaround:
Before migration, lower the max_concurrent_uploads setting in server.conf to 2.

After migration, revert the setting to the default of 8.

Workaround:
Create a $SPLUNK_HOME/etc/system/local/user-seed.conf and restart Splunk

[user_info]
PASSWORD = <yourpassword>

Workaround:
Splunk believes this was introduced in a Microsoft Windows patch. The workaround is to configure a delayed start of the Splunk service(s) so that it starts after the Windows Event Log service.

Workaround:
Use bin/splunk createssl server-cert -d etc/auth/ -n server to generate a new certificate.

Workaround:
Upgrade Hadoop to 2.8.2 or higher.

Workaround:
Reenable the bucket copy saved search and let it run, or force the archiving to happen via | archivebuckets force=1 and then rerun the search.

Workaround:
Set minsplits=maxsplits

Workaround:
Add the journal.gz to the input path's blacklist (vix.input.1.ignore = ....)

Workaround:
To mitigate this issue, make sure that vix.splunk.home.hdfs (or Working directory in the UI) is unique on both search heads that are not in a pool. To keep your instances in the same working directory, configure vix.splunk.search.cache.path to be unique on both search heads.