Keeping Elasticsearch Byte-Balanced at Scale

August 23, 2026

The short version

1. Our 3-node Elasticsearch cluster drifted to 91%/94%/95% disk usage even though shard counts looked even across nodes
2. The default balance.disk_usage weight is so weak that the allocator effectively ignores byte distribution when shard counts are balanced
3. Setting it 10× higher triggered immediate relocations and converged disk usage to within 1% across all nodes

The problem: shard-balanced ≠ byte-balanced

Our prod cluster runs three nodes, all carrying both master and data roles. Each node has a 4100Gi PVC. Nodes were sitting at 91%, 94%, and 95% disk usage. 95 would constanlty trigger alerts.

The shard count dashboard looked fine. Even distribution across nodes.Elasticsearch’s allocator reports balance in terms of shard count, not bytes.

The allocator uses a weighted scoring function with three main components:

The disk_usage weight is four orders of magnitude smaller than the shard and index weights. In practice, it only influences decisions when everything else is perfectly tied. If your indices vary significantly in size - some with large historical shards, some with many small ones - the allocator will happily keep the cluster “shard-balanced” while bytes pile up unevenly.

In our case, the drift was ~190 GB spread across nodes. Shard counts were even. The allocator had no incentive to move anything.

The fix

One persistent cluster setting:

PUT _cluster/settings
{
  "persistent": {
    "cluster.routing.allocation.balance.disk_usage": "2e-10"
  }
}

This is 10× the default. It raises byte-balance to a weight the allocator actually acts on relative to shard count and per-index balance.

After applying, the allocator began relocating shards within minutes. No manual intervention is needed. The cluster converged from 91%/94%/95% to 93%/93%/94% - roughly 60 GB spread across nodes instead of ~190 GB. Good enough given the index size variance we have.

To revert if needed:

PUT _cluster/settings
{
  "persistent": {
    "cluster.routing.allocation.balance.disk_usage": null
  }
}

Setting to null removes the persistent override and returns to the default.

How the allocator actually runs

One thing worth understanding: the allocator is reactive, not scheduled. It doesn’t run on a timer. It reconsiders placement when something changes:

• A new index is created or rolled over
• A node restarts or joins the cluster
• An index is deleted
• A shard assignment fails and is retried

If the cluster is completely idle - no rollovers, no restarts - the allocator won’t trigger even if nodes are unbalanced. The setting change alone doesn’t immediately kick off relocations; it’s the next triggering event that does.

In our case, there was enough ongoing rollover activity that relocations started quickly. If your cluster is quiet, you can nudge it manually:

POST _cluster/reroute?retry_failed=true

Concurrent shard moves are capped by cluster.routing.allocation.cluster_concurrent_rebalance, which defaults to 2. That’s intentional - relocating too many shards simultaneously adds recovery I/O that can degrade search performance. We left it at 2.

Bonus: orphaned indices from snapshot restore

While investigating the disk situation we found something else: 48 indices matching data-import-6.6.2-2024.* with no ILM policy attached, accounting for roughly 200 GB across the cluster.

These indices had a creation_date of the same day in mid-2024 - all of them. That’s not organic index creation; that’s a snapshot restore. And that’s the problem: snapshot restore does not re-apply index templates.

When these indices were originally created, an index template would have attached the 30-day data-import-* ILM policy automatically. When they were restored from snapshot, they came back as-is - no template application. They’re stranded. They’ll sit there indefinitely unless someone deletes them manually.

We confirmed they were safe to delete by checking access patterns - 47 total search queries across all 48 indices over 9 days, zero document fetches. Only metadata operations (Kibana index pattern resolution, _count, size-zero searches). No application was reading actual documents from them.

Safe deletion process:

# Step 1: close the indices - any caller gets an explicit error instead of silent miss
POST /data-import-6.6.2-2024.*/_close

# Step 2: wait 7–10 days, watch for complaints

# Step 3: snapshot to GCS before deleting

# Step 4: delete
DELETE /data-import-6.6.2-2024.01.*,data-import-6.6.2-2024.02.*

Closing first is the right move. It turns any accidental dependency into a loud, visible error rather than a silent data miss.

ILM policy versioning gotcha. Separately, watch out for this: if you edit an ILM policy to remove a phase action (like rollover), indices that were already mid-lifecycle under the old policy can get stuck. They were created expecting a rollover action; the policy no longer has one; they retry the step forever. Check any index that looks stalled:

GET /data-import-*/_ilm/explain

Look for step stuck on a phase that no longer exists in the current policy. The fix is usually to manually advance the index to the next step or reassign a corrected policy.

When it breaks

Disk usage still drifting despite even shard counts.

The default balance.disk_usage weight is too weak to overcome shard-count and per-index weights. Raise it:

PUT _cluster/settings
{
  "persistent": { "cluster.routing.allocation.balance.disk_usage": "2e-10" }
}

Then trigger a rebalance event (rollover, reroute, or wait for the next natural trigger).

Setting applied but no relocations happening.

The allocator is reactive - it needs a triggering event. Force one:

POST _cluster/reroute?retry_failed=true

ILM policy shows indices stuck in a phase that no longer exists.

You edited the policy and removed a step that existing indices were waiting on. Use ILM explain to identify which step they’re stuck on, then either move them forward manually or reassign the updated policy and retry.

Found indices with no ILM policy and no recent access.

Check if they were restored from a snapshot - snapshot restore doesn’t re-apply index templates or ILM policies. Verify access patterns with the stats API (_stats/search,indexing), close the indices, wait for complaints, then delete.

One node consistently higher than the others after rebalancing.

Your balance.disk_usage weight may need to go higher, or you have a few very large shards that can’t be split. Check the largest shards:

GET _cat/shards?v&s=store:desc&h=index,shard,prirep,store,node

If a single shard is significantly larger than others, that shard can’t be split by the allocator - it’ll always count as one unit regardless of its byte size.