the annotation "service-upstream" is not inherited by the "canary" ingress.

[andyliuliming]

What happened:
in this document:
https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#known-issues
all the annotations should be ignored (actually inherrited from the main ingress).

but for the "service-upstream" annotation, if we use the "service-upstream": true in the main ingress.
and for the canary ingress, we do not specify it. then from the output of the
/dbg backends all

the backends are still the node ip/ports. not the service one.

could you please help check whether this is by design or just one bug?

What you expected to happen:

NGINX Ingress controller version (exec into the pod and run /nginx-ingress-controller --version):

Kubernetes version (use kubectl version):

Environment:

• Cloud provider or hardware configuration:

• OS (e.g. from /etc/os-release):

• Kernel (e.g. uname -a):

• Install tools:

  • Please mention how/where was the cluster created like kubeadm/kops/minikube/kind etc.

• Basic cluster related info:

  • kubectl version
  • kubectl get nodes -o wide

• How was the ingress-nginx-controller installed:

  • If helm was used then please show output of helm ls -A | grep -i ingress
  • If helm was used then please show output of helm -n <ingresscontrollernamespace> get values <helmreleasename>
  • If helm was not used, then copy/paste the complete precise command used to install the controller, along with the flags and options used
  • if you have more than one instance of the ingress-nginx-controller installed in the same cluster, please provide details for all the instances

• Current State of the controller:

  • kubectl describe ingressclasses
  • kubectl -n <ingresscontrollernamespace> get all -A -o wide
  • kubectl -n <ingresscontrollernamespace> describe po <ingresscontrollerpodname>
  • kubectl -n <ingresscontrollernamespace> describe svc <ingresscontrollerservicename>

• Current state of ingress object, if applicable:

  • kubectl -n <appnamespace> get all,ing -o wide
  • kubectl -n <appnamespace> describe ing <ingressname>
  • If applicable, then, your complete and exact curl/grpcurl command (redacted if required) and the reponse to the curl/grpcurl command with the -v flag

• Others:

  • Any other related information like ;
    • copy/paste of the snippet (if applicable)
    • kubectl describe ... of any custom configmap(s) created and in use
    • Any other related information that may help

How to reproduce this issue:

Anything else we need to know:

[andyliuliming]

not sure whether that's only a display issue.
maybe the endpoints of the backend won't be used when the noserver is true?

as set in:

ingress-nginx/internal/ingress/controller/controller.go

Line 1036 in 23bd743

upstreams[defBackend].NoServer = true

[strongjz]

Can you test that if you set the service-upstream on the carnary ingress, it does set the service IP instead of node IP/port?

[strongjz]

/triage needs-information

[hsubramanianaks]

@strongjz Yes it does set the clusterIP if we set service-upstream: true on the canary ingress. Please let me know if you need more information. we are wondering why the annotations on stable is not inherited to canary ingress.

[strongjz]

Carnary is for testing different versions of an application and possible different configurations, so this seems to be by design. We discussed this on the community call and wouldn't consider this a bug if it works if you set it directly on the canary.

[hsubramanianaks]

@strongjz we were wondering if we copy over the annotations from stable to canary to maintain consistency and avoid conflicts/different behaviors. The controller wouldn't cause any issues ex: conflicts if it tries to inherit stable ingress annotations in any flow , causing failures ?

[andyliuliming]

@strongjz do you know who maintains this document?
https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#known-issues
just want to try to understand this section:

"Note that when you mark an ingress as canary, then all the other non-canary annotations will be ignored (inherited from the corresponding main ingress) except nginx.ingress.kubernetes.io/load-balance, nginx.ingress.kubernetes.io/upstream-hash-by, and annotations related to session affinity. If you want to restore the original behavior of canaries when session affinity was ignored, set nginx.ingress.kubernetes.io/affinity-canary-behavior annotation with value legacy on the canary ingress definition."

from this section, I think the annotation for the canary will inherit from the corresponding main ingress. but looks like this is not the current behaivour, at least the "server-upstream" is not "inherited".

I'm thinking whether the "noserver": true make it happen, or something else? please help.

[longwuyuan]

/remove-kind bug

[longwuyuan]

/kind support

[strongjz]

Everyone on the project is responsible for keeping the documentation updated. Unfortunately, that section of the docs has not been updated in 8 years, so it's likely to be outdated. The majority of that code has not been updated for 2 to 7 years, so this very well may be a bug.

IIUC service-upstream is not a canary-related annotation so that it will be ignored on a canary ingress and not inherited.

[andyliuliming]

++ @ElvinEfendi who contributed to this document in this commit:
#3477
Hi @ElvinEfendi, could you please help?
cc @aledbf

pr which implemented the canary feature FYI:
#3341
cc @clandry94