Requirements of distributions for configuration

[zeitlinger]

To figure out how distributions can maintain their functionality with file config, let's collect some use cases, which can be added as comments to this issue.

Based on that, we can create a prototype implementation.

@jack-berg FYI

[zeitlinger]

Grafana distribution for Java: https://github.com/grafana/grafana-opentelemetry-java

1. add a metric view based on a grafana specific property (to enable a feature in the UI): https://github.com/grafana/grafana-opentelemetry-java/blob/a4309356da4d162a140e6f8aeb770e3374615956/custom/src/main/java/com/grafana/extensions/filter/ServerAddressConfig.java
2. limit enabled instrumentations for support purposes: https://github.com/grafana/grafana-opentelemetry-java/blob/8ddb5ceeed775cd45510bcfa57ccd906cb74662f/custom/src/main/java/com/grafana/extensions/instrumentations/TestedInstrumentationsCustomizer.java
3. truncate resource attributes - to align with the backend: https://github.com/grafana/grafana-opentelemetry-java/blob/50411a43bf63896d4f68f1634d5a047e365399d2/custom/src/main/java/com/grafana/extensions/resources/ResourceCustomizer.java#L21
4. add a server timing header that can't currently be configured in any way: https://github.com/grafana/grafana-opentelemetry-java/blob/e01f090c6338df900b19eaa98c250b093a34fa03/custom/src/main/java/com/grafana/extensions/servertiming/ServerTimingHeaderActivator.java#L15
5. add some default properties: https://github.com/grafana/grafana-opentelemetry-java/blob/cb31391d38030cba9d8aa4871c4fad179e557be2/custom/src/main/java/com/grafana/extensions/GrafanaAutoConfigCustomizerProvider.java#L29-L31

It's basically 2 types of changes:

• A: things that can't currently be configured via properties (3, 4)

• B: things that can be configured (at least with file config) - but that are cumbersome (1, 2, 5)

• "A" should not interfere with the file config mechanism, since the distro doesn't alter what the user configures in yaml

• "B" should

  • always respect when the user overrides the defaults that the distro sets
  • should output the produced yaml somehow (e.g. on debug log)

[zeitlinger]

@jeanbisutti also wants to help with understading the needs of the MS distro for Java

[jeanbisutti]

@jeanbisutti also wants to help with understading the needs of the MS distro for Java

cc @trask

[jeanbisutti]

@zeitlinger As discussed, the Microsoft distribution of the OpenTemetry Java agent provides a JSON file allowing the user to configure OTel and non-OTel features (that could potentially be implemented on the OTel side).

Some specific configurations:

• Configure distribution logging
• Enable log telemetry from a log level
• Sampling override
• Telemetry processor

cc @trask @mattmccleary

[jack-berg]

Hi all - moving to the spec repo since this repo is just concerned with the declarative configuration data model. The spec would be the place that would specify any common hooks for distributions to customize the declarative config.

[jack-berg]

It's basically 2 types of changes:

There's a third type of change which is things that can't be configured via declarative config. Consider OTLP proxy configuration, which is available via the opentelemetry-java programmatic config API (i.e. the OTLP exporter builders), but which which is not (yet) available in declarative config. If a distribution wants to configure these types of things, they need some sort of hook / callback to access the various SDK components as they're being constructed. In opentelemetry-java, AutoConfigurationCustomizer facilitates this.

We might want to say that this is out of scope, and instead encourage distributions to extend declarative configuration to make sure it covers all the properties they need.

---

Declarative config has a design principle of "what you see is what you get". If a user specifies their config in a declarative config YAML file, only to have the resolved SDK be different, then that is confusing and erodes trust.

This principle is in conflict with distributions having hooks / callbacks to customize config contents.

I can see a couple of paths forward:

• If SDKs to support customization hooks, then they MUST log a warning indicating to the user that the config is being customized. Additionally, there SHOULD be options to print out the configuration model after customization is applied.
• Reject SDK customization hooks, and instead encourage distributions to build abstractions which modify configuration contents upstream of the SDK. For example, a distribution could have an initialization script which reads / modifies a declarative configuration contents. Or, a vendor could use helm to provide a curated configuration experience, allowing the user to toggle a pre-determined set of options, which helm uses to resolve a YAML configuration file.

---

I wonder if distribution customization requirements are language agnostic, or are mostly specific to opentelemetry-java and the tendency for vendors to publish distributions of the otel java agent? This is a great issue to collect feedback and fine out.

[zeitlinger]

It's basically 2 types of changes:

There's a third type of change which is things that can't be configured via declarative config.

I think this is the same as my "A: things that can't currently be configured via properties (3, 4)"

We might want to say that this is out of scope, and instead encourage distributions to extend declarative configuration to make sure it covers all the properties they need.

That would be a show-stopper - at least for the Grafana distribution.

Declarative config has a design principle of "what you see is what you get". If a user specifies their config in a declarative config YAML file, only to have the resolved SDK be different, then that is confusing and erodes trust.

I agree - and I think we can find a solution that meets that bar

This principle is in conflict with distributions having hooks / callbacks to customize config contents.

• if distros customize what can already be customized (e.g. by setting defaults), then we should
  • make sure that the user has the final say (if they change the default, it's respected)
  • make sure that the changes are easily understandable - e.g. by outputting the effective yaml
• if distros add a completely new thing (e.g. by using programmatic config, some java agent hooks, new instrumentations), then there should be a new config element in a vendor namespace that makes those settings visible
  • those settings could again have defaults, which are outputted in the same way as defaults for standard settings

• If SDKs to support customization hooks, then they MUST log a warning indicating to the user that the config is being customized. Additionally, there SHOULD be options to print out the configuration model after customization is applied.

yes, this is basically what I proposed above

• Reject SDK customization hooks, and instead encourage distributions to build abstractions which modify configuration contents upstream of the SDK.

this basically defeats one of the main reasons to have a distribution - to have features before they are accepted upstream

• For example, a distribution could have an initialization script which reads / modifies a declarative configuration contents. Or, a vendor could use helm to provide a curated configuration experience, allowing the user to toggle a pre-determined set of options, which helm uses to resolve a YAML configuration file.

we should not build a feature based on the presence of tools like helm - that's too restricting

I wonder if distribution customization requirements are language agnostic, or are mostly specific to opentelemetry-java and the tendency for vendors to publish distributions of the otel java agent? This is a great issue to collect feedback and fine out.

I think it's mostly a java thing - but I'm also looking for more feedback

[zeitlinger]

Apart from distributions, we should also take into account the use cases from the java agent and spring starter.

• change default protocol in agent: https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/f6176919e4a0fa194bc7aa747984c3ac09a64604/javaagent-tooling/src/main/java/io/opentelemetry/javaagent/tooling/config/OtlpProtocolPropertiesSupplier.java#L17-L20
• agent tracer: https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/3122897b2e0eb4d9dfbf27404ef232f97278ae84/javaagent-tooling/src/main/java/io/opentelemetry/javaagent/tooling/AgentTracerProviderConfigurer.java#L28
• resource provider config in agent: https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/9e83898f0daed944b1ff958351d7452cd235cd67/sdk-autoconfigure-support/src/main/java/io/opentelemetry/instrumentation/resources/ResourceProviderPropertiesCustomizer.java#L20
• spring boot resource adapter: https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/081415e6b1868e223481a932358e668ac00713de/instrumentation/spring/spring-boot-autoconfigure/src/main/java/io/opentelemetry/instrumentation/spring/autoconfigure/OpenTelemetryAutoConfiguration.java#L96-L108

IIUC, all of those use cases could be implemented as a YamlStructuredConfigProperties to YamlStructuredConfigProperties transformation.

[zeitlinger]

here's a first PoC to try the configuration: grafana/grafana-opentelemetry-java#896

It uses OpenTelemetryConfigurationModel rather than YamlStructuredConfigProperties, because that seems to be easier to get right - it already has the full parsed structure.

Filling this structure with the required spring boot properties will be a challenge - but that's the next step.