[AWS Firehose] populate event.dataset field for ingested records

[Kavindu-Dodan]

Background

Firehose integration ¹ utilizes several other AWS data streams added through AWS integration ². Most of those integrations do not currently define event.dataset field (ex: see Cloudtrail fields³).

Problem

Other components rely on the existence of the event.dataset field. For example, consider the pre-build security rule ⁴. These rules will fail to work with current integration configurations as data lacks the required field.

Solution

Update integrations to add event.dataset where possible with correct constant values. For example, Cloudtrail should have event.dataset: aws.cloudtrail. This must get added through relevant AWS assets as Firehose internally perform rerouting ( for example see logs ⁵ and metrics ⁶)

Footnotes

1. https://github.com/elastic/integrations/tree/main/packages/awsfirehose ↩

2. https://github.com/elastic/integrations/tree/main/packages/aws ↩

3. https://github.com/elastic/integrations/blob/main/packages/aws/data_stream/cloudtrail/fields/base-fields.yml ↩

4. https://www.elastic.co/guide/en/security/current/aws-iam-login-profile-added-to-user.html ↩

5. https://github.com/elastic/integrations/blob/main/packages/awsfirehose/data_stream/logs/routing_rules.yml ↩

6. https://github.com/elastic/integrations/blob/main/packages/awsfirehose/data_stream/metrics/routing_rules.yml ↩