Why is CPU Usage Increasing After Setting Resource Limits for Airbyte Container-Orchestrator Pods?

[karolur]

Hello Airbyte Community,

I would appreciate any help in understanding how the container-orchestrator resources work. I have Airbyte deployed in GKE using Helm chart 0.248.5 and Airbyte version 0.63.4.

Here are my Helm values:

global:
jobs:
resources:
requests:
memory: 600Mi
limits:
memory: 1Gi

With these values I noticed that the container orchestrator pods were using 100% to 110% of CPU, while the rest of the pods were at about 40% utilization. Each container orchestrator pod uses around 800 to 900 millicores of CPU, so I figured I could add a request of 1 CPU and a limit of 2 CPU to bring the CPU utilization down to around 85%.

I updated the values to:

jobs:
resources:
requests:
cpu: 1000m
memory: 600Mi
limits:
cpu: 2000m
memory: 1Gi

However, after making this change, the container orchestrator pods started using 1900m to 2100m of CPU, causing the CPU utilization to increase to 200% instead of decreasing to 85%.

I'm puzzled by this behavior, I even attempted to remove the values and putting them again just to make sure that was what was causing this.

How do resource requests and limits affect the actual CPU usage in the container orchestrator? Any insights or explanations would be greatly appreciated.

Thank you!