Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[8.17](backport #41636) [libbeat] fix: aws & openstack metadata conflict in add_cloud_metadata processor #42606

Merged
merged 1 commit into from
Feb 5, 2025

Conversation

mergify[bot]
Copy link
Contributor

@mergify mergify bot commented Feb 5, 2025

Proposed commit message

This PR fixes incorrect recognition of EC2/AWS cloud provider as Openstack. The root cause was the common metadata endpoints used by both AWS SDK & Openstack logic. And this happened when IMDSv2 is disabled in AWS.

I attempted to migrate Openstack logic to another metadata implementation. However, I did not manage to create a fully functioning setup to validate the implementation. Hence, this PR focuses on a priority-based solution where priority is given for SDK-backed metadata fetching over HTTP endpoints.

Current priory providers are - aws/ec2 & azure

Note - I have done a minor refactoring to rename Local struct property to DefaultEnabled to make intention clearer

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

You need a local copy and an EC2 instance to validate the fix.

  • Enable metadata service in EC2 instance and make IMDSv2 optional
  • Build a beats (ex:- metricbeat) module based on this libbeat change
  • Copy the beats module to EC2 instance and start the module with add_cloud_metadata processor enabled & logs set to debug for more in-depth logs & no provider enforced
  • Observe logs and see data (ex:- system monitoring) through Kibana to validate cloud provider detection

Related issues

Screenshots

-IMDSv2 disabled

Screenshot 2024-11-13 at 11 45 38 AM

  • Processor enabled but no provider enforced

Screenshot 2024-11-13 at 11 50 24 AM

  • Debug logs on multi-result and priority based selection

Screenshot 2024-11-13 at 11 44 47 AM

  • Cloud provider detected correctly,

Screenshot 2024-11-13 at 11 45 13 AM


This is an automatic backport of pull request #41636 done by [Mergify](https://mergify.com).

…a processor (#41636)

* rename misleading variable

Signed-off-by: Kavindu Dodanduwa <[email protected]>

* introduce provider priority

Signed-off-by: Kavindu Dodanduwa <[email protected]>

# Conflicts:
#	libbeat/processors/add_cloud_metadata/providers.go

* isolate priority logic and add testing

Signed-off-by: Kavindu Dodanduwa <[email protected]>

* documentation

Signed-off-by: Kavindu Dodanduwa <[email protected]>

* review changes

Signed-off-by: Kavindu Dodanduwa <[email protected]>

---------

Signed-off-by: Kavindu Dodanduwa <[email protected]>
(cherry picked from commit 6d4e641)
@mergify mergify bot added the backport label Feb 5, 2025
@mergify mergify bot requested a review from a team as a code owner February 5, 2025 16:43
@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Feb 5, 2025
@Kavindu-Dodan Kavindu-Dodan added the Team:obs-ds-hosted-services Label for the Observability Hosted Services team label Feb 5, 2025
@elasticmachine
Copy link
Collaborator

Pinging @elastic/obs-ds-hosted-services (Team:obs-ds-hosted-services)

@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Feb 5, 2025
@Kavindu-Dodan Kavindu-Dodan enabled auto-merge (squash) February 5, 2025 16:46
@Kavindu-Dodan Kavindu-Dodan merged commit cf5c18e into 8.17 Feb 5, 2025
143 checks passed
@Kavindu-Dodan Kavindu-Dodan deleted the mergify/bp/8.17/pr-41636 branch February 5, 2025 18:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport Team:obs-ds-hosted-services Label for the Observability Hosted Services team
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants