Configurable Sidecar Container Name

[s-bauer]

Please describe the Improvement and/or Feature Request
Currently, the injected sidecar envoy proxy always has the name "envoy", it is hardcoded as a constant in your code. This can lead to issues with other solutions that are using the same name ("envoy") as their container name. It would be great if you can add a config value that allows this name to be changed.

Scope (please mark with X where applicable)

• New Functionality []
• Install [ ]
• SMI Traffic Access Policy [ ]
• SMI Traffic Specs Policy [ ]
• SMI Traffic Split Policy [ ]
• Permissive Traffic Policy [ ]
• Ingress [ ]
• Egress [ ]
• Envoy Control Plane [ ]
• CLI Tool [ ]
• Metrics [ ]
• Certificate Management [ ]
• Sidecar Injection [x]
• Logging [ ]
• Debugging [ ]
• Tests [ ]
• CI System [ ]
• Demo [ ]
• Project Release [ ]

[phillipgibson]

@s-bauer can you provide more details into the specific use case. I understand that someone could potentially deploy a container with the name envoy, but under what circumstances would that be? The envoy container happens for deployments in a managed OSM namespace that will get the side-car injection. We have not seen a pattern of application containers being named "envoy" in a service mesh. I would like to gather more information on that scenario.

[s-bauer]

Im facing this issue in combination with Bridge to Kubernetes where they are having a routingmanager, which is an envoy instance with the container name being "envoy". That leads to the mentioned collision when sidecar injection is enabled for this workload/namespace. Having a properly unique name for the injected sidecar (e.g. "osm-envoy") or having it configurable would allow this conflict to be resolved.

[phillipgibson]

I'll need to look into the Bridge to K8s config, but does the namespace for that solution need to be managed by OSM? If you have a link to the config of your setup I'll take a look.

[s-bauer]

Oh and I think it's the same issue with contour (when deploying that separately from osm by using the gateway provisioned). I've resided to not inject a sidecar for contour which might be the better idea anyways.

[s-bauer]

I'll need to look into the Bridge to K8s config, but does the namespace for that solution need to be managed by OSM? If you have a link to the config of your setup I'll take a look.

So Bridge to Kubernetes, if you're not aware, is replacing a workload in k8s and redirecting all the traffic or only part of the traffic to your local machine. This allows you to debug a service locally easily. They have two modes:

• no-isolation: replaces the service in k8s and redirects ALL traffic to your local machine
• isolation: deploys the routingmanager (envoy) in k8s, that redirects most traffic to the original service and part of it (based on a HTTP header) to your local machine.

In general, they only support service mesh with the no-isolation mode. But having a quick look at their architecture makes me believe even the isolation mode should work fine with a service mesh if configured correctly. Unfortunately, due to the naming conflict with the "envoy" container, testing this became a bit more difficult. Because both, osm and BridgeToKubernetes, have big involvement from Microsoft, it might be a good idea to team up with them and properly integrate with OSM :)

[phillipgibson]

So I've had a chance to look at the architecture for Bridge to K8s and from what I can see the issue extends beyond the container name and is not specific to OSM, but to all service meshes that are injecting a proxy for the pod. Even if you had the ability to rename the pod from the mesh, (you can also check and see if Bridge to K8s support container naming for their proxy as well) I'm not sure how the routing would actually work with the additional proxy. We faced something similar with integrating Dapr with OSM because Dapr utilizes sidecars too. I would reach out to the Bridge to K8s team and file an issue for service mesh support because again this is beyond just the container name collision in the pod. There looks to be functionality needed in the Routing Manager Pod as well as some policy on mTLS egress from the agent. Remember once the service mesh proxy is injected, the IP tables are programmed to route all traffic to the service mesh proxy. It looks like there is significant header mutation happening for their proxies to route traffic and without the correct header configurations it will not work.

[s-bauer]

After quite a bit of troubleshooting and try and error, I finally got it working. I'll drop some info into the Bridge to Kubernetes github project to help them add proper support for osm. Still, I think it would be a good idea for osm to add a configuration option to change the name of the injected sidecar. No matter the specific use-case, the name "envoy" is not specific enough and might always cause conflicts!

While I have you (@phillipgibson) here, maybe you can answer me another quick question: I'm not able to access pods directly when osm is installed. I can ping them just fine, but no HTTP traffic is coming through. I assume this is expected behavior with OSM, but I'm wondering if there's any configuration option to change that. This was one of the issues with Bridge to Kubernetes that I've faced. They are trying to access the pod directly by it's IP, but that is failing. Creating a service to expose that pod and then accessing that one works fine.

[s-bauer]

In case anyone is interested, these are the changes required in BridgeToKubernetes to support running in an OSM enabled namespace: Azure/Bridge-To-Kubernetes#221

As I assumed, no big changes in the architecture were necessary. All components involved in BridgeToKubernetes are just like any other application running on a k8s cluster. No special low level networking or anything is happening. Just a very basic Envoy proxy that is routing to two different destinations based of a header value.

[phillipgibson]

After quite a bit of troubleshooting and try and error, I finally got it working. I'll drop some info into the Bridge to Kubernetes github project to help them add proper support for osm. Still, I think it would be a good idea for osm to add a configuration option to change the name of the injected sidecar. No matter the specific use-case, the name "envoy" is not specific enough and might always cause conflicts!

While I have you (@phillipgibson) here, maybe you can answer me another quick question: I'm not able to access pods directly when osm is installed. I can ping them just fine, but no HTTP traffic is coming through. I assume this is expected behavior with OSM, but I'm wondering if there's any configuration option to change that. This was one of the issues with Bridge to Kubernetes that I've faced. They are trying to access the pod directly by it's IP, but that is failing. Creating a service to expose that pod and then accessing that one works fine.

That experience would be expected as the authz needed for mTLS is based on the service name on the originator and destination. The cert isn't signed with IP information.

[s-bauer]

Yea I figured it's something like this. So direct pod-to-pod communication will never be possible with a service mesh (or more specifically OSM)?

[github-actions]

This issue will be closed due to a long period of inactivity. If you would like this issue to remain open then please comment or update.

[github-actions]

Issue closed due to inactivity.