Metric nomad.client.allocs.memory.cache always 0 since Nomad 1.9.0

[SCORP111]

Follow-up to #24524

Nomad version

1.9.0 to 1.9.5

Operating system and Environment details

Ubuntu 24.04 with cgroupsv1 (testet in a VM and in WSL2)

Issue

The value of the metric nomad.client.allocs.memory.cache is always 0 since Nomad version 1.9.0

Reproduction steps

Start nomad in dev-mode with config that has telemetry enabled: sudo nomad agent -dev -config=config.hcl

config.hcl:

telemetry {
  collection_interval = "1s"
  publish_allocation_metrics = true
  publish_node_metrics       = true
  prometheus_metrics = true
}

Run job thats consuming some RAM:

job "mysql" {
  datacenters = ["dc1"]

  group "mysql" {
    network {
      port "http" {
        static = "5678"
      }
    }
    task "server" {
      driver = "docker"

      env {
        MYSQL_ROOT_PASSWORD = secret
        MYSQL_ALLOW_EMPTY_PASSWORD = true
        MYSQL_RANDOM_ROOT_PASSWORD = false
      }

      config {
        image = "mysql"
        ports = ["http"]
      }

      resources{
        memory = 1024
      }
    }
  }
}

Expected Result

nomad.client.allocs.memory.cache should output a non-zero-value like this:

curl 127.0.0.1:4646/v1/metrics | jq | grep -A1 cache

"Name": "nomad.client.allocs.memory.cache",
"Value": 5763072.0

Actual Result

Nomad version 1.9.0:

curl 127.0.0.1:4646/v1/metrics | jq | grep -A1 cache

"Name": "nomad.client.allocs.memory.cache",
"Value": 0.0

Nomad version 1.9.5:

curl 127.0.0.1:4646/v1/metrics | jq | grep -A1 cache

"Name": "nomad.client.allocs.memory.cache",
"Value": 0.0

Nomad version 1.8.0:

curl 127.0.0.1:4646/v1/metrics | jq | grep -A1 cache

"Name": "nomad.client.allocs.memory.cache",
"Value": 5844992.0

Nomad version 1.8.1:

curl 127.0.0.1:4646/v1/metrics | jq | grep -A1 cache

"Name": "nomad.client.allocs.memory.cache",
"Value": 10321920.0

Nomad version 1.8.4:

curl 127.0.0.1:4646/v1/metrics | jq | grep -A1 cache

"Name": "nomad.client.allocs.memory.cache",
"Value": 9773056.0

Thanks for your help!

[Juanadelacuesta]

Hello @SCORP111 Thank you for reporting this bug, there was a big update of the docker driver done in the 1.9.x releases and unfortunately some things, like metrics, were affected by it, but we are working hard to bring everything back to normal.
I will move this to our TODOs list and hopefully it will be picked up soon.