Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

output loki remove_keys in nest key not work properly #9091

Closed
owanio1992 opened this issue Jul 16, 2024 · 3 comments · May be fixed by #10013
Closed

output loki remove_keys in nest key not work properly #9091

owanio1992 opened this issue Jul 16, 2024 · 3 comments · May be fixed by #10013
Labels
Stale status: waiting-for-triage

Comments

@owanio1992
Copy link

owanio1992 commented Jul 16, 2024 •
edited
Loading

Bug Report

can't not set multiple remove_keys in a nest key

Describe the bug
sample data

   {
      "time""=>""2024-07-16T03:25:43.552908882Z",
      "stream""=>""stderr",
      "_p""=>""F",
      "log""=>""2024-07-16 03:25:43,552 INFO     werkzeug 10.42.3.1 - - [16/Jul/2024 03:25:43] "GET / HTTP/1.1" 200 -",
      "kubernetes""=>"{
         "pod_name""=>""data-coll-nbl-exporter-5c89f6c5f9-mfqb5",
         "namespace_name""=>""devconn",
         "pod_id""=>""7b3883ec-c05a-49b3-b506-a10d21092151",
         "labels""=>"{
            "app.kubernetes.io/instance""=>""data-coll",
            "app.kubernetes.io/managed-by""=>""Helm",
            "app.kubernetes.io/name""=>""nbl-exporter",
            "app.kubernetes.io/version""=>""main",
            "date""=>""1721099472",
            "helm.sh/chart""=>""nbl-exporter-0.1.0",
            "pod-template-hash""=>""5c89f6c5f9"
         },
         "annotations""=>"{
            "prometheus.io.scheme""=>""http",
            "prometheus.io/path""=>""/metric_redis_llen",
            "prometheus.io/port""=>""50000",
            "prometheus.io/scrape""=>""true"
         },

   }

case1 , it's can work

remove_keys kubernetes['namespace_name']

case2 , it's can work

remove_keys kubernetes['pod_id']

case3 , only pod_id remove

remove_keys kubernetes['namespace_name'],kubernetes['pod_id']

case4 , only namespace_name remove

remove_keys kubernetes['pod_id'],kubernetes['namespace_name']

Expected behavior
all specific key will be remove

Your Environment

  • Version used: 3.1.2
  • Environment name and version : Kubernetes 1.29
  • Filters and plugins:
    fliter: kubernetes
@fivetran-aleksandarkostic
Copy link

fivetran-aleksandarkostic commented Aug 15, 2024 •
edited
Loading

I am facing the same issue, keys referenced as either labels or structured_metadata are not being removed from kubernetes map automatically, and it is also impossible to remove them manually as shown in this issue.
I believe that the underlying issue is sorting as described in this comment: #5546 (comment) as I was able to get different keys like namespace_name or container_name to be removed automatically when changing the ordering in label-map.json, but not all of them.

I tested latest (3.1.6) version of Fluent-bit and the issue is still there.

@github-actions GitHub Actions
Copy link
Contributor

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

@github-actions github-actions bot added the Stale label Nov 14, 2024
@github-actions GitHub Actions
Copy link
Contributor

This issue was closed because it has been stalled for 5 days with no activity.

@github-actions github-actions bot closed this as not planned Won't fix, can't repro, duplicate, stale Nov 19, 2024
@shirolimit shirolimit mentioned this issue Feb 26, 2025
Open
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Stale status: waiting-for-triage
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

mp: fix flb_mp_accessor_keys_remove for sibling keys removal shirolimit/fluent-bit
2 participants
@owanio1992 @fivetran-aleksandarkostic