In Grafana Explore, or by editing any visualizations on any Dashboard, you can display the top reported metrics using the following PromQL Query:
topk(10, count by (__name__)({__name__=~".+"}))
This query finds the 10 metrics with the highest cardinality.
If you want, you can track more labels like this:
topk(10, count by (__name__, component)({__name__=~".+"}))
Next, change the query Type
to Instant
.
Your metrics cardinalities are likely not changing over time, so we just need a snapshot of the current counts, and not a graph of metrics and their cardinalities over time.
When you’re done, click Run query
in the top right corner of your screen. You should see the a table with metrics and their corresponding cardinalities
From here, you can query any individual high-cardinality metric to drill down into all its different permutations.
For example, the apiserver_request_duration_seconds_bucket
metric on my cluster had >6k different label combinations, some of them where not interesting and where dropped at scrape time on Prometheus side.
NOTE: Ensure that query Type is still set to Instant or your query may time out.