Changes to Github OAuth flow for PKCE?

[mig5]

Hi Vouch team!

I am using Vouch with Github as the IDP. Today I got this email from Github:

This is a notice that your apps need to be updated to be compatible with a new security feature on github.com.

GitHub added PKCE support (RFC 7636) for the OAuth flow this week - see this changelog announcement from Monday. While rolling out the feature, we found that some applications were already using PKCE, but in a way that was incompatible with GitHub’s implementation - including yours.

You have been granted a 60 day exemption to update use of PKCE during user sign-in for the following apps:
• GitHub Apps: N/A
• OAuth Apps: (masked)
Please validate your application is correctly using PKCE:

1. On the authorization leg, include both the code_challenge_method and code_challenge query parameters. 1. code_challenge_method must be S256. We do not support plain.
2. The code_challenge must be the base64 Url_encoding of the SHA-256 hash of the code_verifier.
3. On the authorization code exchange leg, include the code_verifier that was used to generate the code_challenge. The verifier must be at least 43 characters.

To use PKCE, your app must send both authorization leg parameters and must send the code_verifier on the authorization code exchange leg. Your app was exempted from this rollout due to finding one or more of the following issues:

• Not sending the code_verifier after beginning authentication with a challenge and method.
• Including only one of code_challenge_method or code_challenge but not both.
• Sending an invalid code_challenge – it must be 43 characters in length.
• Using a plain code challenge method instead of S256.
• Sending an invalid code_verifier that did not match the code_challenge or was too short.

To update your application, please ensure that the above issues are resolved. You can create a new application to test your authentication flow to ensure that it works. To learn more about GitHub’s support for PKCE, see our developer documentation:
• Authenticating users with GitHub Apps

• Authenticating with OAuth apps

---

I checked your code and also my config, and found that S256 is indeed already the default code challenge method for Github.

I also confirmed that the code_challenge is 43 characters.

Based on the list above of possible reasons, it would then imply one or both of:

• Not sending the code_verifier after beginning authentication with a challenge and method.
• Sending an invalid code_verifier that did not match the code_challenge or was too short.

So I'm not clear on what else needs to change? Based on the above, should I change something in my config, or (I suspect) is this a deeper change that needs to happen in Vouch itself to support the Github PKCE flow?

Thanks for any advice!

[mig5]

I actually wonder if they sent this in error... I just created a new Github OAuth2.0 app and it worked, but I hadn't changed any of my config other than the Client ID and Client Secret, and that I added callback_url which I hadn't previously set and yet it had been working..

Based on what they said above, it implies the new app is using the new method, and it worked fine... maybe it was me adding the callback_url but that wasn't one of the requirements they listed. At this page they say that sending the redirect_uri is 'strongly recommended' but not that without it it will break PKCE, but perhaps it does..

Sorry for the noise..

[aaronpk]

If I understand the message they sent, they sent that to developers who at some point had sent a PKCE code_challenge value that was not in the correct format. Do you think there's a chance you had at some point used the client ID/secret with a test app that did PKCE wrong? Vouch has never sent the code_challenge to GitHub.

[mig5]

@aaronpk Hmm, I don't think I did. Vouch introduced PKCE defaulting to S256 for Github in 2020, I am pretty sure I started using Vouch with Github after that in the usual way.

I'm now seeing a separate strange thing. I restarted my Vouch app after testing adding, removing, and then re-adding, the callback_url parameter to my config. Now I'm getting a 400 error in Vouch:

https://vouch.example.com/auth/xxxxxxxxx/?code=yyyyyyyyyyyyy&state=zzzzzzzzzz

vouch | {"level":"warn","ts":1752625939.0730953,"msg":"/auth Error while retrieving user info after successful login at the OAuth provider: oauth2: "bad_verification_code" "The code passed is incorrect or expired." "https://docs.github.com/apps/managing-oauth-apps/troubleshooting-oauth-app-access-token-request-errors/#bad-verification-code\""}

This is weird..

[aaronpk]

oh funny, you're right, I see Vouch is adding a code_challenge on the request to GitHub. I don't actually see where Vouch is configured to use PKCE for Github, can you point me to that?

[mig5]

@aaronpk see

vouch-proxy/handlers/login.go

Lines 277 to 280 in 66ba3ef

	if cfg.GenOAuth.CodeChallengeMethod != "" {
	opts = append(opts, oauth2.SetAuthURLParam("code_challenge_method", cfg.GenOAuth.CodeChallengeMethod))
	opts = append(opts, oauth2.SetAuthURLParam("code_challenge", session.Values["codeChallenge"].(string)))
	}

and see

vouch-proxy/handlers/login.go

Lines 298 to 313 in 66ba3ef

	func appendCodeChallenge(session sessions.Session) {
	var codeChallenge string
	var CodeVerifier, _ = cv.CreateCodeVerifier()
	switch strings.ToUpper(cfg.GenOAuth.CodeChallengeMethod) {
	case "S256":
	codeChallenge = CodeVerifier.CodeChallengeS256()
	break
	case "PLAIN":
	codeChallenge = CodeVerifier.CodeChallengePlain()
	// TODO support plain text code challenge
	log.Fatal("plain code challenge method is not supported")
	return
	default:
	log.Fatal("Code challenge method %s is invalid", cfg.GenOAuth.CodeChallengeMethod)
	return
	}

and here is where it defaults to S256 for Github:

vouch-proxy/pkg/cfg/oauth.go

Line 270 in 66ba3ef

GenOAuth.CodeChallengeMethod = "S256"

[aaronpk]

well this is very weird...

I've been adding log statements in various parts of the code to trace it, always getting back the same error:

Error while retrieving user info after successful login at the OAuth provider: oauth2: "invalid_grant" "A code_verifier was not included, but the authorization request included a code_challenge. Codes requested with a code_challenge MUST be redeemed with a code_verifier." "https://docs.github.com/apps/creating-github-apps/authenticating-with-a-github-app/generating-a-user-access-token-for-a-github-app\""}

Finally GitHub threw up a warning dialog saying there have been an unusual number of requests from this application, and required that I click "continue". After that, Vouch successfully got an access token.

When I tried again, when I didn't see the interrupt screen, it went back to the same error message!

[aaronpk]

Weirdly sometimes the error message I get is this:

Error while retrieving user info after successful login at the OAuth provider: oauth2: "bad_verification_code" "The code passed is incorrect or expired." "https://docs.github.com/apps/managing-oauth-apps/troubleshooting-oauth-app-access-token-request-errors/#bad-verification-code\""}

In any case, it seems to be consistently reproducible regardless of which error message is returned. Every 10th attempt gives me the interstitial prompt and the token request after that prompt works.

[mig5]

Thanks.. I am persistently seeing that last error you saw, on a new Github OAuth2.0 client. The only times I didn't, were:

1. The first time I used the Github OAuth2.0 app (and approved the consent screen), and
2. When I turned on testing: true in Vouch and I manually clicked a link on the Vouch page for something like this:

https://vouch.example.com/auth/{STATE-IS_HERE}/?code={CODE VERIFIER WAS HERE?}&state={STATE-IS_HERE}

What was interesting was that there were two such links in the testing page, with different codes, and the one I clicked worked... might be a clue.

Every 10th attempt gives me the interstitial prompt and the token request after that prompt works.

I wonder if it is in effect recreating the initial 'consent' page, which since it worked for me the first time after consent, might be why it then works then too.

At least I'm not going crazy! Thanks for debugging with me :)

[aaronpk]

So if we've narrowed it down to it working when there is an interstitial, maybe it is some sort of race condition inside of Vouch which is broken when there is a long enough gap between the authorization request and the token request...

[mig5]

For anyone who can work on this (it's beyond my skill, I don't know golang but I know OIDC pretty well) - I turned on debug logging and the 'testing: true' mode. Here is the log output of what happens:

https://gist.github.com/mig5/8c46d5f3eadf126104772866c6db0981

I've just masked the client_id, code_challenge, state and code attributes to aaaaaaaa, bbbbbbbbb, ccccccccc and ddddddddddd respectively, but it should hopefully make sense.

Unfortunately I'm not seeing too much there about how Vouch sends the code to Github to trigger that 400 response. Maybe it's too low-level for the debug logs.

We've been given a 60 day window by Github to fix our implementation, if it can't be done in time then I guess I will have to change my IDP to something else than Github, at least temporarily.

[mig5]

When I revoke my OAuth2.0 user token with my OAuth2.0 app in Github (to force the consent again), I am surprised to see in the Vouch debug screen that there are two journeys to https://github.com/login/oauth/authorize, each with different code_challenges and states (as shown here, first with bbbbbbbbbbbbbbb and then with dddddddddddddddddd:

All 302 redirects will be captured and presented as links here

    [login](https://vouch.example.com/login)
    [logout](https://vouch.example.com/logout)
    [validate](https://vouch.example.com/validate)
    https://example.com/
    https://github.com/login/oauth/authorize?client_id=aaaaaaaaaaa&code_challenge=bbbbbbbbbbbbbbbbb&code_challenge_method=S256&redirect_uri=https%3A%2F%2Fvouch.example.com%2Fauth&response_type=code&scope=read%3Auser&state=cccccccccccccccccccc
    [/auth/cccccccccccccccccccc/?code=zzzzzzzzzzzzzz&state=cccccccccccccccccccc](https://vouch.example.com/auth/cccccccccccccccccccc/?code=zzzzzzzzzzzzzz&state=cccccccccccccccccccc)
    https://github.com/login/oauth/authorize?client_id=aaaaaaaaaaa&code_challenge=dddddddddddddddddd&code_challenge_method=S256&redirect_uri=https%3A%2F%2Fvouch.example.com%2Fauth&response_type=code&scope=read%3Auser&state=eeeeeeeeeeeeeeeeeeeeee
    [/auth/eeeeeeeeeeeeeeeeeeeeee/?code=gggggggggggggg&state=eeeeeeeeeeeeeeeeeeeeee](https://vouch.example.com/auth/eeeeeeeeeeeeeeeeeeeeee/?code=gggggggggggggg&state=eeeeeeeeeeeeeeeeeeeeee)

It's when I click on the second link, it works...

[reinout]

I think I'm seeing the same issue. Since the github pkce announcement, login via vouch fails periodically.
An intermediate "solution" we're using now is to use the "revoke all user tokens" button in our github oauth apps' settings (we have several). A fresh login works, but after a few hours or days one or more users complain about a failing login again. Revoking the tokens makes it work again.

According to the person getting the issue most often: "Failing login" means a 400 page with the vouch proxy logo. "And if I remember correctly, upon a redirect to /auth/.....

Note: this is not my best issue comment, sorry. Normally, I'd provide screenshots and some better information. But I'm off on a laptop-less holiday in a few hours and thought it best to give some feedback or possible extra info. Ignore this comment when not relevant :-)