contrib/internal/httptrace: add support for inferred proxy spans

[jordan-wong]

What does this PR do?

This PR introduces functionality to infer proxy spans based on headers that we instruct users to inject upstream, allowing these headers to be propagated downstream. The headers are then used to infer a proxy span to provide increased visibility to a user's trace. The headers are as follows:

Header Name	AWS Mappings	Header Contents
x-dd-proxy	aws-apigateway	aws-apigateway
x-dd-proxy-request-time-ms	context.requestTimeEpoch	Start time of request in ms
x-dd-proxy-domain-name	context.domainName	AWS domain name
x-dd-proxy-httpmethod	context.httpMethod	HTTP method used for request
x-dd-proxy-path	context.path	Path for request
x-dd-proxy-stage	context.stage	AWS stage name

Motivation

There is no current support for API Gateway and other proxies that do not run Datadog code. By inferring spans via headers, the gaps in observability can be filled to provide a more accurate trace.

Reviewer's Checklist

• Changed code has unit tests for its functionality at or near 100% coverage.
• System-Tests covering this feature have been added and enabled with the va.b.c-dev version tag.
• There is a benchmark for any new code, or changes to existing code.
• If this interacts with the agent in a new way, a system test has been added.
• Add an appropriate team label so this PR gets put in the right place for the release notes.
• Non-trivial go.mod changes, e.g. adding new modules, are reviewed by @DataDog/dd-trace-go-guild.
• For internal contributors, a matching PR should be created to the v2-dev branch and reviewed by @DataDog/apm-go.

Unsure? Have a question? Request a review!

[zarirhamza]

Needs a bit of cleanup and logging for prod but logically looks good. Going to request @wantsui to look at it as well to confirm the span information is relatively similar as in dd-trace-js.

[pr-commenter]

Benchmarks

Benchmark execution time: 2025-01-22 22:12:35

Comparing candidate commit 39087f1 in PR branch add-inferred-proxy-spans with baseline commit 241a6f1 in branch main.

Found 0 performance improvements and 3 performance regressions! Performance is the same for 55 metrics, 1 unstable metrics.

scenario:BenchmarkStartRequestSpan-24

• 🟥 allocated_mem [+72 bytes; +72 bytes] or [+40.909%; +40.909%]
• 🟥 allocations [+1; +1] or [+20.000%; +20.000%]
• 🟥 execution_time [+66.057ns; +67.123ns] or [+21.642%; +21.992%]

[zarirhamza]

Let's also make sure that we test that our inferred span matches the behavior of the parent span by asserting things like that status code are the same

[datadog-datadog-prod-us1]

Datadog Report

Branch report: add-inferred-proxy-spans
Commit report: 64501fb
Test service: dd-trace-go

✅ 0 Failed, 5207 Passed, 72 Skipped, 2m 19.94s Total Time

[darccio]

@zarirhamza @jordan-wong Do you know why the benchmarks are reporting this 50%+ increase in execution time in BenchmarkStartRequestSpan?

🟥 execution_time [+159.899ns; +160.941ns] or [+52.326%; +52.667%]

[darccio]

@zarirhamza @jordan-wong I think this unconditional extraction of spanParentCtx is the source of the performance impact.

Would it be possible to isolate everything related to the inferred proxy spans so it runs if the feature is activated?

[zarirhamza]

Previously, the span extraction would happen on every call anyway just within the span option func. I tried isolating all behaviors as much as possible so that should improve the performance.

[zarirhamza]

Also unsure if the fact we now have cleanup functions via StartRequestSpans may result in increased usage that is expected solely due to behavior

[darccio]

Looks like the increased CPU time and memory are to be expected due to the new implementation.