"Search Not Executed": Fixing the Minimum Free Disk Space Error After a Splunk ES Upgrade
You just upgraded Splunk Enterprise Security. The upgrade wizard finished clean, the app version looks right, and then an analyst tries to run a search and gets this:
Search not executed: The minimum free disk space (5000MB) reached for
/opt/splunk/var/run/splunk/dispatch. user=jdoe, concurrency_category="historical",
concurrency_context="user_instance-wide", current_concurrency=1, concurrency_limit=5000
At first glance this message points you in two wrong directions at once. It mentions concurrency, so it looks like a search scheduling problem. It mentions the dispatch directory, so it looks like a dispatch artifact problem. In most cases it is neither. It is a full disk, and after an ES upgrade there is a very specific reason why.
What the Error Actually Means
Splunk checks free disk space on the partition that holds the dispatch directory every time it launches a search. If free space falls below the minFreeSpace threshold, which defaults to 5000MB and lives in server.conf under the [diskUsage] stanza, splunkd refuses to dispatch any new search. Scheduled searches, correlation searches, ad hoc searches from the UI, all of them stop.
The concurrency fields in the message are just context about the search that was rejected. current_concurrency=1 with a limit of 5000 tells you concurrency was never the problem. The operative words are "minimum free disk space reached."
This is a protective behavior, not a bug. Splunk would rather block searches than let dispatch artifacts fill the volume completely and take down splunkd, the KV Store, or the OS itself.
Step 1: Confirm the Disk Is Actually Full
df -h /opt/splunk
In our case:
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/rhel-opt 100G 95G 4.9G 96% /opt
4.9G available, threshold is 5000MB. The volume crossed the line and Splunk did exactly what it is configured to do. Now the question becomes: what ate the disk?
Step 2: Find the Bloat
Do not guess. Measure:
du -xh --max-depth=2 /opt/splunk | sort -rh | head -25
On a search head, you would normally expect the big consumers to be under var (dispatch artifacts, logs, KV Store, maybe local summaries). Here is what we found instead:
93G /opt/splunk
60G /opt/splunk/etc
36G /opt/splunk/etc/apps
25G /opt/splunk/etc/backup
18G /opt/splunk/var
8.3G /opt/splunk/var/log
That 60G under etc is the tell. On a healthy search head, etc is usually somewhere between 1G and 5G. Config directories do not grow to 60G on their own. Something put data there, and the timing of the ES upgrade is not a coincidence.
The Real Culprit: /opt/splunk/etc/backup
When you upgrade Splunk Enterprise or install a major app upgrade like ES, Splunk snapshots your existing configuration into /opt/splunk/etc/backup before making changes. The ES installer does the same with the previous version of the app suite. This is a sensible safety net.
The problem is that nothing ever cleans it up. Each upgrade adds another full snapshot, and on an ES search head those snapshots are large because ES ships dozens of supporting apps, lookup files, and threat intel content. Ours had accumulated 25G of backups nobody had touched since they were written.
The fix:
ls -lh /opt/splunk/etc/backup/
Identify what is in there. If your upgrade is validated and stable, copy the most recent backup off-box if you want a safety net, then remove the rest:
rm -rf /opt/splunk/etc/backup/<old_backup_name>
The moment free space climbs back above the threshold, searches dispatch again. No restart required, because the disk check happens at search dispatch time, not at startup.
In our case, clearing the backup directory took the volume from 96% to roughly 70% and the error disappeared immediately.
Other Places to Check
The backup directory was the headline, but an ES upgrade can inflate several other locations. Work through these before you declare victory:
Dispatch directory. ES runs a heavy scheduled search load: datamodel accelerations, correlation searches, threat intel downloads. Artifacts can pile up, especially if TTLs are generous or searches were orphaned during the upgrade. Clean them the supported way rather than deleting by hand:
/opt/splunk/bin/splunk cmd splunkd clean-dispatch /opt/splunk/dispatch_old 4h
This relocates artifacts older than the given age to a directory you specify. Verify nothing is missed, then delete the relocated copy. Avoid raw rm -rf inside a live dispatch directory.
etc/apps. Look for stale versioned copies of apps (SplunkEnterpriseSecuritySuite.old, hand-made .bak directories), installer tarballs someone extracted and forgot, and oversized lookup files. Asset and identity CSVs and threat intel lookups are the usual suspects on an ES search head:
du -sh /opt/splunk/etc/apps/* | sort -rh | head -15
var/log/splunk. Upgrades that hit errors can make python.log, web_service.log, or splunkd.log balloon. A multi-gigabyte log is worth reading before you rotate it away, because it may be describing something that half-broke during the upgrade:
ls -lhS /opt/splunk/var/log/splunk | head -10
KV Store. ES leans on the KV Store for threat intel collections and tracking data. Check du -sh /opt/splunk/var/lib/splunk/kvstore. If it is huge, prune through the application (intel retention settings), not the filesystem.
Old diags and crash dumps. diag-*.tar.gz files in the Splunk home or crash-*.log files in the log directory quietly accumulate.
The Stopgap You Should Avoid If You Can
You can lower the threshold in server.conf:
[diskUsage]
minFreeSpace = 2000
This works, but understand what you are trading. The 5000MB default exists because a search head that runs completely out of disk fails much less gracefully than one that stops dispatching searches. KV Store corruption, truncated config writes, and splunkd crashes are all worse outcomes. Treat threshold changes as a bridge to a real fix, not the fix.
Preventing the Repeat
Three takeaways from this incident:
1. Add backup cleanup to your upgrade runbook. After every validated upgrade: archive the snapshot in /opt/splunk/etc/backup somewhere off the search head, then delete it locally. Splunk will never do this for you, and each skipped cleanup compounds into the next disk incident.
2. Size ES search heads with upgrade headroom. A 100G volume for an ES search head is tight. Between dispatch churn, KV Store growth, log volume, and the fact that every upgrade temporarily needs room for a full config snapshot plus the new app bundle, 300G or more is a much safer floor. If you are on LVM, extending is usually a low-risk lvextend plus xfs_growfs with no downtime, so get the request in before the next upgrade window rather than during the next outage.
3. Read the error past its first line. The concurrency fields in this message send a lot of engineers down the wrong path, adjusting search limits on a system whose actual problem is a full disk. When a Splunk error names a directory and a size threshold, check the filesystem before you check the configuration.
Quick Reference
| Symptom | Check | Fix |
|---|---|---|
| "minimum free disk space reached" on all searches | df -h /opt/splunk | Free space above minFreeSpace (default 5000MB) |
| Disk full right after ES/Splunk upgrade | du -sh /opt/splunk/etc/backup | Archive off-box, delete local snapshots |
etc/apps abnormally large | du -sh /opt/splunk/etc/apps/* | sort -rh | Remove stale app copies, tarballs, prune giant lookups |
| Dispatch directory bloated | du -sh /opt/splunk/var/run/splunk/dispatch | splunk cmd splunkd clean-dispatch <dest> <age> |
| Logs ballooned during upgrade | ls -lhS /opt/splunk/var/log/splunk | Read for root cause, then rotate |
The error cleared for us within seconds of freeing the space, with no restart. If yours does not, confirm you freed space on the same filesystem that hosts the dispatch path, since var/run occasionally lives on a separate mount.