Can't get Flux GitHub auth working with CDK EKS Blueprints

[JamesMcMahon]

Hey Folks,

Trying to get Flux working using CDK and an add-on for the EKS blueprints. I can not get auth to work at all and keep getting:

GitRepository/redacted failed to checkout and determine revision: unable to clone 'https://github.com/redacted': authorization failed

A lot of the Flux docs are centered around flux bootstrap but I can't use that since I am using CDK to manage my cluster setup. The documentation for it the add-on suggests using a bearer token, but doing some research it seems like you can't clone unless you basic access auth. So I switched over, but still no luck.

Secret resource for reference:

apiVersion: v1
data:
  password: <redacted>
  username: <redacted>
immutable: false
kind: Secret
metadata:
  name: flux-github-credentials
  namespace: flux-system
  <trimming out runtime annotations etc>
type: Opaque

The repository and source controller hasn't given me more than the message above. Any suggestions for debugging further?

[JamesMcMahon]

I am just realizing now that when I switched over to basic access auth the errors went from needs authentication to needs authorization so I double checked and low and behold I forget to set my personal access token to my organization versus my personal space.

Just for the future I'd love to know if there was a way to get deeper output. Additionally knowing how to force the source controller to resync would have been helpful because it started going into some kind of backoff mode where it was only rechecking my creds even 20 minutes or so.

[stefanprodan]

There are many commands available in the Flux CLI for operating Flux e.g.

flux get all -A
flux reconcile 
flux check
flux events
flux logs

[JamesMcMahon]

Thanks Stefan.

Additionally knowing how to force the source controller to resync

I think flux reconcile source is what I was looking for here.

Just for the future I'd love to know if there was a way to get deeper output.

Maybe flux logs would have helped here - I view the Kubernetes logs of the various Flux resources while debugging, not sure if logs would give me more. The situation is resolved now, so I can no longer verify without breaking things again. It would have been nice to know the auth method of source controller specifically because it switches behavior (as far as I understand from the docs) dynamically based on the shape of the secret.

[JamesMcMahon]

The two things I'd like to do when I get time is to update the CDK blueprint docs so they are not recommending bearer tokens for Github access with Flux and update the Flux docs with some clearer messaging around bearer tokens not working. There is already a message in the docs but it made me think that bearer tokens would work in some instance but that doesn't seem to be the case.

But don't know when I will get a chance to submit those.

[stefanprodan]

There is no --token-auth option in the CDK blueprint?

[stefanprodan]

Ah I figured it out, the issue is that GitHub does not support bearerToken, you need to setup a secret with basic auth and set the GitHub PAT as the password and git as the username. Docs here: https://fluxcd.io/flux/components/source/gitrepositories/#basic-access-authentication

[JamesMcMahon]

Yeah that was the page I was looking at a lot yesterday 😄

Note: If you are looking to use OAuth tokens with popular servers (e.g. GitHub, Bitbucket, GitLab), you should use basic access authentication instead. These servers use basic HTTP authentication, with the OAuth token as the password. Check the documentation of your Git server for details.

This line was making me think that in some cases you could use bearer tokens - didn't realize my personal access token was an OAuth token. I may submit a PR to make that language a little clearer that those services don't work with bearer tokens. But honestly looking at it now I think I was just being dense.

git as the username

I actually tried that and it didn't work, but it may have been due to missing authorization on the token. What I ended up with was just putting my Github username in there.

[stefanprodan]

You also need to change the GitRepository URL to https from ssh

[JamesMcMahon]

https is actually working fine for me - for SSH don't you need https://fluxcd.io/flux/components/source/gitrepositories/#ssh-authentication?

Or you are saying to use the git username you need the SSH url?