Tracking: Stabilize declarative configuration

[jack-berg]

We've made a lot of progress in declarative configuration, and I'd like to discuss what scope should be included in an initial cut at stabilization, and blockers.

The declarative config spec is broken into a variety of pieces:

• Data model: SDK implementation requirements around data model, and definition of schema. Subcomponents:
  • opentelemetry-configuration: repository for data model JSON schema
  • file-based YAML representation, including env var substitution syntax
  • SDK in-memory representation
• Instrumentation configuration API: defines an API for instrumentation to participate in configuration. Subcomponents:
  • ConfigProperties for programmatic access of structured config
  • ConfigProvider for accessing instrumentation config
• SDK: SDK user facing capabilities. Subcomponents:
  • ConfigProvider SDK implementation of ConfigProvider API component
  • ComponentProvider defines mechanism for referencing / loading SDK extension plugin components (i.e. exporters, samplers, etc) in declarative config
  • Parse and Create operations for producing configured SDK components from YAML file representation
• OTEL_EXPERIMENTAL_CONFIG_FILE, for automatic initialization based on YAML file representation

I propose the following scope for initial stability:

• opentelemetry-configuration
• file-based YAML representation
• SDK in-memory representation
• ConfigProperties
• ComponentProvider
• Parse and Create operations
• OTEL_EXPERIMENTAL_CONFIG_FILE, adjusted to OTEL_CONFIG_FILE

That means the following would be out of scope:

• ConfigProvider (i.e. instrumentation config API)

Assuming this scope, the follow lists the known blockers to stability

• Cut stable 1.0.0 release opentelemetry-configuration#161 (see list of sub-issues)
• Use ABNF syntax instead of PCRE2 in file configuration env var substitution syntax #4027
• Alignment with OpenTelemetry Collector configuration #3963
  • [file configuration] Escaping environment variable substitution #3914

The following list are not specifically blocking, but they should be reviewed to ensure the stability of declarative configuration doesn't prevent us from addressing them in the future:

• Environment variables specification should require _FILE variations to support reading from files #2739

Additionally, we need evaluation / feedback of the state of the spec from the current prototype language implementations.

If you have feedback on the scope of initial stability, or on additional blockers, please discuss in this issue.

[codeboten]

Not strictly blocking, but we should ensure we're not preventing addressing issues like this one #2739

[jack-berg]

#4416 was opened and its worth discussing whether its blocking for stability. Currently, the declarative config spec accommodates customization via language in OTEL_EXPERIMENTAL_CONFIG_FILE:

Implementations MAY provide a mechanism to customize the configuration model parsed from OTEL_EXPERIMENTAL_CONFIG_FILE.

My opinion is that this language is flexible enough for opentelemetry-java and any other implementation that has customization requirements to build out tooling. Although, as a maintainer of opentelemetry-java, I would hesitate to make such tooling stable without more discussion in the spec, since if there is going to be more standardization around how customization works I would want opentelemetry-java to conform to that.

I personally don't consider #4416 blocking since: 1. Declarative config is still quite useful without it. 2. There is nothing in the spec that prevents future enhancement in this area. I do think #4416 is a very interesting area as a followup to stabilization.

Interested in what others think, including @open-telemetry/configuration-approvers.