Control Metric Cardinality Before It Kills Your Time-Series Database

September 17, 2026

The short version

Our metrictank tank.metrics_active alert kept firing above 2M series and resolving every 30 minutes. The sawtooth pattern was normal - metrictank’s staleness sweep runs every 2 hours - but the peak was climbing day-over-day. Something was genuinely growing.

1. Querying the live metrictank index to investigate OOMKilled the pod. Twice.
2. The safe path: read directly from Cassandra using mt-index-cat scoped to a single partition
3. Three metric path patterns accounted for ~68% of all active series - all caused by per-tenant identifiers embedded in the metric path
4. Short-term fix: tighter max-stale prune rules. Long-term fix: tags, not path segments

The system

We run metrictank as our time-series aggregation layer, backed by Cassandra on a dedicated VM outside Kubernetes. The index holds ~15M persisted series total. tank.metrics_active measures the subset actively receiving data - the number metrictank has to keep hot in memory. Our alert threshold was 2M.

The sawtooth is expected behaviour: metrictank’s internal staleness sweep periodically evicts series that haven’t received a datapoint recently, dropping the active count, then it climbs again as data flows in. What’s not expected is the ceiling rising every day.

The wrong first move

The obvious thing to do when you want to know what’s in the index is ask the index. Metrictank exposes /metrics/index.json for exactly this purpose.

We curl’d it from inside the pod. The pod OOMKilled.

We tried again with wget to a tmp file, thinking streaming to disk would help. The pod OOMKilled again.

The endpoint serializes the entire in-memory index - all ~15M series - before writing a single byte of response. At that scale it’s hundreds of megabytes of JSON. There’s no pagination, no streaming, no filter that applies before serialization. Querying it on a large index is just a slow way to kill your own server.

Rule we wrote down afterward: never query the live daemon for cardinality investigations. Query the storage layer directly.

The safe investigation path

Metrictank ships a CLI tool called mt-index-cat that reads the index directly from Cassandra, bypassing the running server entirely. We run it as a one-off Kubernetes Job using the same metrictank image.

There’s a critical detail about how mt-index-cat works: it does not stream. It calls cassandra-idx.Load() and pulls the entire index into an in-memory trie first, then iterates over it. At ~700 bytes per entry and 15M entries, a full load needs roughly 10 GB of memory. The -max-stale flag only narrows what gets printed - it does nothing to limit what gets loaded.

The solution is -partitions. Cassandra stores the metrictank index across 16 partitions (0–15). Scoping to one partition loads ~1M entries, which fits comfortably in 2–3 Gi. For a cardinality investigation you don’t need all 16 - one partition gives you a representative 1/16th sample, enough to identify dominant patterns.

command: ["/bin/sh", "-c"]
args:
- |
  mt-index-cat \
    -partitions 0 \
    cass -hosts cassandra-host:9042 -timeout 30s -keyspace metrictank \
    '{{println .Name}}' \
  2>/dev/null
resources:
  requests: { memory: 2Gi, cpu: "1" }
  limits:   { memory: 4Gi }
nodeSelector: { preemptible: 'true' }

One gotcha: the Go text/template syntax {{"\n"}} triggers an “unterminated quoted string” parse error in this image’s template parser. Use {{println .Name}} instead.

Stream the output locally as soon as the pod reaches Running - don’t try to aggregate inside the container:

kubectl logs -f <job-pod> > sample.txt

Aggregating with awk inside the container on a 15M-row output OOM’d at 4Gi. Do it locally:

awk -F. '{print $1"."$2"."$3}' sample.txt | sort | uniq -c | sort -rn | head

This groups by the first three path segments - enough to identify which services and environments are responsible without drowning in the full metric name.

What we found

From a single-partition sample (~156k series, extrapolated to ~2M active):

Three patterns, ~68% of all active series.

The common thread across all of them: per-tenant identifiers embedded directly in the metric path. Names like company_NNN and pot_NNN where NNN is a customer or account ID. When you combine a tenant segment with per-table, per-measurement, and per-stat dimensions, each new tenant doesn’t add a few series - it adds a full copy of every table × measurement × stat combination. The fan-out is multiplicative.

This is a well-known Graphite/metrictank footgun. Tags were introduced specifically to avoid this pattern. But migrating instrumentation is slow, and in the meantime the index just keeps growing.

The fix

Short-term: tighter prune rules.

Without an explicit match, every series falls through to metrictank’s [default] index rule, which has a max-stale of 60 days. A metric that stops being emitted - because a tenant churns, a table is dropped, a job is deleted - still occupies the active index for 60 days before it’s pruned. For services that regularly churn tenants or tables, that 60-day window fills up fast.

We added two targeted rules:

• High-throughput service reporter metrics: max-stale = 3d
• Postgres table-stats metrics: max-stale = 7d

Shorter windows mean stale series from churned tenants expire quickly instead of accumulating over two months.

Stopgap: raise the alert threshold from 2.0M to 2.3M while the prune rules propagate through the index sweep cycle. The rules don’t evict immediately - they take effect on the next staleness sweep pass.

Long-term: upstream instrumentation change.

Prune rules treat the symptom. The root cause is that per-tenant identifiers don’t belong in the metric path at all. They belong as tags. service.latency;tenant=NNN produces one series regardless of how many tenants you have; service.NNN.latency produces one series per tenant, forever.

This is a conversation with service owners, not a config change. The payoff is significant - the top two patterns alone are 50% of the active index - but it requires updating instrumentation and migrating dashboards.

When it breaks

Alert firing: tank.metrics_active above threshold.

First, check the shape. A clean sawtooth that resolves within the sweep cycle is expected behaviour. A sawtooth whose ceiling is rising week-over-week means the active subset is genuinely growing - investigate the index.

Need to inspect what’s in the index.

Do not hit /metrics/index.json on the live server. Run mt-index-cat as a Kubernetes Job scoped to a single Cassandra partition (-partitions 0). Stream logs to a local file, aggregate locally with awk. See the job spec above.

mt-index-cat job OOMKills.

You forgot -partitions. Without it the job loads all 16 partitions - ~10 GB. Add -partitions 0 (or any single partition number 0–15) to bound memory to ~2–3 Gi.

Found the top offenders, want to prune them.

Add named rules in your index-rules config with regex patterns matching the offending prefixes and set max-stale to match the data’s actual useful lifetime - not the 60-day default. Prune rules only take effect on the next staleness sweep pass, so raise the alert threshold temporarily if the sweep cycle is long.

Prune rules are in place but the count isn’t dropping.

The sweep hasn’t run yet, or the pattern regex isn’t matching. Verify the regex against a sample series name from your mt-index-cat output before assuming the rule is working.

Series count keeps growing despite prune rules.

The churn rate exceeds the prune rate - new tenants or tables are being added faster than old ones expire. This is the signal that the upstream instrumentation needs to change: per-tenant path segments need to become tags.