feat(DRIVERS-2903): use custom aws configuration

[durran]

Adds section to auth spec and CSFLE spec on allowing user provided custom credential providers for AWS and notes on testing.

Node implementation: mongodb/node-mongodb-native#4373

• Update changelog.
• Test changes in at least one language driver.
• Test these changes against all server versions and topologies (including standalone, replica set, sharded
  clusters, and serverless).

[jyemin]

Just a few quick edits while I perused this.

[jyemin]

Since this needs to be supported in FLE as well, I think we need to update the FLE spec and specify how to configure the "aws" kms provider. Consider adding a similar property with same semantics to AWSKMSOptions in https://github.com/mongodb/specifications/blob/master/source/client-side-encryption/client-side-encryption.md#kmsproviders.

(While in some drivers you might be able to apply an auth mechanism property to FLE, in Java that isn't possible as auth mechanism properties are associated with a credential, not the MongoClient, and the credential of the MongoClient might not even be an AWS credential (or could be a different one).

[durran]

Since this needs to be supported in FLE as well, I think we need to update the FLE spec as well and specify how to configure the "aws" kms provider. Consider adding a similar property with same semantics to AWSKMSOptions in https://github.com/mongodb/specifications/blob/master/source/client-side-encryption/client-side-encryption.md#kmsproviders.

(While in some drivers you might be able to apply an auth mechanism property to FLE, in Java that isn't possible as auth mechanism properties are associated with a credential, not the MongoClient, and the credential of the MongoClient might not even be an AWS credential (or could be a different one).

@kevinAlbs Do you have thoughts on this? Would there be a case where auto-encryption could want a different AWS credential provider than the associated MongoClient. (With ClientEncryption this is already possible since it has its own MongoClient)

[baileympearson]

@durran What about users who aren't using AWS authentication for their clients but want to use AWS automatic KMS credential fetching with a custom provider? I don't think Node would support that without a provider option for KMS specifically.

[baileympearson]

@durran What about users who aren't using AWS authentication for their clients but want to use AWS automatic KMS credential fetching with a custom provider? I don't think Node would support that without a provider option for KMS specifically.

[durran]

@durran What about users who aren't using AWS authentication for their clients but want to use AWS automatic KMS credential fetching with a custom provider? I don't think Node would support that without a provider option for KMS specifically.

That's a good point and I think a valid use case. I'll update everything accordingly.

[blink1073]

cc @kevinAlbs since this affects Client Encryption options

[baileympearson]

Suggested change

	aws? AWSCredentialProvider
	aws?: AWSCredentialProvider

[baileympearson]

Is it sufficient to test that the data key is successfully created? Credentials can already be fetched from the environment, so even if a driver didn't use the custom credential provider I'd expect this test to pass.

Is there a way we could make this test fail if the credential provider isn't used? In Node, we can always spy on the provider but I don't think every language can do this. Maybe we could manually implement spying by having the provider increment a call count, and then assert that callCount = 1. Or we could use environment variables that don't match what the AWS sdk expects:

# these have the wrong names, aws sdk looks for ACCESS_KEY_ID and SECRET_ACCESS_KEY
process.env = {
  KEY: <key id>,
  SECRET: <secret>
}

const provider: AWSCredentialProvider = async () => {
  return {
    accessKeyId: process.env.KEY,
    secretAccessKey: process.env.SECRET
  }
}

(While this would work in Node, where we can re-write process.env as we wish, but I'm not sure this would be possible in other languages without a separate evergreen task that didn't load the AWS credentials into the environment.).

[baileympearson]

I have the same concern for the regular auth tests too.

[baileympearson]

Do you think we should test this case for auto encryption as well? Most (all?) drivers have separate code paths for instantiating a CE object and instantiating a client configured for auto encryption. This test only covers explicit encryption.