MSI on Azure ML network unreachable

[sbatten]

Core Library

MSAL Node (@azure/msal-node)

Core Library Version

3.2.3

Wrapper Library

Not Applicable

Wrapper Library Version

4.7.0

Public or Confidential Client?

Confidential

Description

We have recently upgraded our @azure/identity package from 4.3.0 to 4.7.0 which we use in Azure ML with a user assigned managed identity to fetch key vault credentials. The managed identity auth is now failing with network error. Reverting the version bump fixes the issue, but we cannot do this for other reasons.

Error Message

No response

MSAL Logs

azure:core-client:warning The baseUri option for SDK Clients has been deprecated, please use endpoint instead.
azure:core-client:warning The baseUri option for SDK Clients has been deprecated, please use endpoint instead.
azure:core-client:warning The baseUri option for SDK Clients has been deprecated, please use endpoint instead.
azure:core-rest-pipeline retryPolicy:info Retry 0: Attempting to send request 2ff1e8f6-fda2-42ab-b341-631e9c93487f
azure:keyvault-secrets:info Request: {
"url": "https://vaultname.vault.azure.net/secrets/secretname/?api-version=7.5",
"headers": {
"accept": "application/json",
"accept-encoding": "gzip,deflate",
"user-agent": "azsdk-js-keyvault-secrets/4.8.0 core-rest-pipeline/1.10.1 Node/v20.18.1 OS/(x64-Linux-5.15.0-1079-azure)",
"x-ms-client-request-id": "2ff1e8f6-fda2-42ab-b341-631e9c93487f"
},
"method": "GET",
"timeout": 0,
"disableKeepAlive": false,
"streamResponseStatusCodes": {},
"withCredentials": false,
"tracingOptions": {
"tracingContext": {
"_contextMap": {}
}
},
"requestId": "2ff1e8f6-fda2-42ab-b341-631e9c93487f",
"allowInsecureConnection": false,
"enableBrowserStreams": false
}
azure:core-rest-pipeline:info No cached TLS Agent exist, creating a new Agent
azure:keyvault-secrets:info Response status code: 401
azure:keyvault-secrets:info Headers: {
"cache-control": "no-cache",
"pragma": "no-cache",
"content-type": "application/json; charset=utf-8",
"expires": "-1",
"x-ms-keyvault-region": "eastus",
"x-ms-client-request-id": "2ff1e8f6-fda2-42ab-b341-631e9c93487f",
"x-ms-request-id": "51bb9657-b216-4f80-87d6-92a9c3482bba",
"x-ms-keyvault-service-version": "1.9.2103.1",
"x-ms-keyvault-network-info": "conn_type=Ipv4;addr=20.3.30.51;act_addr_fam=InterNetwork;",
"x-content-type-options": "nosniff",
"strict-transport-security": "max-age=31536000;includeSubDomains",
"www-authenticate": "Bearer authorization="https://login.microsoftonline.com/72f988bf-86f1-41af-91ab-2d7cd011db47\", resource="https://vault.azure.net\"",
"date": "Wed, 26 Feb 2025 02:20:40 GMT",
"content-length": "97"
}
azure:core-rest-pipeline retryPolicy:info Retry 0: Received a response from request 2ff1e8f6-fda2-42ab-b341-631e9c93487f
azure:core-rest-pipeline retryPolicy:info Retry 0: Processing 2 retry strategies.
azure:core-rest-pipeline retryPolicy:info Retry 0: Processing retry strategy throttlingRetryStrategy.
azure:core-rest-pipeline retryPolicy:info Retry 0: Skipped.
azure:core-rest-pipeline retryPolicy:info Retry 0: Processing retry strategy exponentialRetryStrategy.
azure:core-rest-pipeline retryPolicy:info Retry 0: Skipped.
azure:core-rest-pipeline retryPolicy:info None of the retry strategies could work with the received response. Returning it.
azure:identity:info ManagedIdentityCredential => getToken() => Using the MSAL provider for Managed Identity.
azure:identity:info ManagedIdentityCredential - Token Exchange => ManagedIdentityCredential - Token Exchange: Unavailable. The environment variables needed are: AZURE_CLIENT_ID (or the client ID sent through the parameters), AZURE_TENANT_ID and AZURE_FEDERATED_TOKEN_FILE
azure:identity:info ManagedIdentityCredential => getToken() => MSAL Identity source: MachineLearning
azure:identity:info ManagedIdentityCredential => getToken() => Calling into MSAL for managed identity token.
azure:identity:info ManagedIdentityCredential => MSAL Node V2 info message: [Wed, 26 Feb 2025 02:20:40 GMT] : [] : @azure/[email protected] : Info - [Managed Identity] ServiceFabric managed identity is unavailable because one or all of the 'IDENTITY_HEADER', 'IDENTITY_ENDPOINT' or 'IDENTITY_SERVER_THUMBPRINT' environment variables are not defined.
azure:identity:info ManagedIdentityCredential => MSAL Node V2 info message: [Wed, 26 Feb 2025 02:20:40 GMT] : [] : @azure/[email protected] : Info - [Managed Identity] AppService managed identity is unavailable because one or both of the 'IDENTITY_HEADER' and 'IDENTITY_ENDPOINT' environment variables are not defined.
azure:identity:info ManagedIdentityCredential => MSAL Node V2 info message: [Wed, 26 Feb 2025 02:20:40 GMT] : [] : @azure/[email protected] : Info - [Managed Identity] Environment variables validation passed for MachineLearning managed identity. Endpoint URI: http://localhost:46809/msi/token/. Creating MachineLearning managed identity.
azure:identity:info ManagedIdentityCredential => MSAL Node V2 info message: [Wed, 26 Feb 2025 02:20:40 GMT] : [] : @azure/[email protected] : Info - [Managed Identity] Adding user assigned client id to the request.
azure:core-rest-pipeline retryPolicy:info Retry 0: Attempting to send request 42e35aff-c26b-4c24-9998-c965011536dc
azure:core-rest-pipeline retryPolicy:info Retry 0: Attempting to send request 42e35aff-c26b-4c24-9998-c965011536dc
azure:core-rest-pipeline:info Request: {
"url": "http://localhost:46809/MSI/token?api-version=2017-09-01&resource=REDACTED&client_id=REDACTED",
"headers": {
"metadata": "REDACTED",
"secret": "REDACTED",
"content-type": "application/x-www-form-urlencoded;charset=utf-8",
"accept-encoding": "gzip,deflate",
"user-agent": "azsdk-js-identity/4.7.0 core-rest-pipeline/1.10.1 Node/v20.18.1 OS/(x64-Linux-5.15.0-1079-azure)",
"x-ms-client-request-id": "42e35aff-c26b-4c24-9998-c965011536dc"
},
"method": "GET",
"timeout": 0,
"disableKeepAlive": false,
"withCredentials": false,
"abortSignal": {},
"requestId": "42e35aff-c26b-4c24-9998-c965011536dc",
"allowInsecureConnection": true,
"enableBrowserStreams": false
}
azure:core-rest-pipeline:info Response status code: 400
azure:core-rest-pipeline:info Headers: {
"content-type": "text/plain; charset=utf-8",
"date": "Wed, 26 Feb 2025 02:20:40 GMT",
"content-length": "60"
}
azure:core-rest-pipeline retryPolicy:info Retry 0: Received a response from request 42e35aff-c26b-4c24-9998-c965011536dc
azure:core-rest-pipeline retryPolicy:info Retry 0: Processing 2 retry strategies.
azure:core-rest-pipeline retryPolicy:info Retry 0: Processing retry strategy throttlingRetryStrategy.
azure:core-rest-pipeline retryPolicy:info Retry 0: Skipped.
azure:core-rest-pipeline retryPolicy:info Retry 0: Processing retry strategy exponentialRetryStrategy.
azure:core-rest-pipeline retryPolicy:info Retry 0: Skipped.
azure:core-rest-pipeline retryPolicy:info None of the retry strategies could work with the received response. Returning it.
azure:core-rest-pipeline retryPolicy:info Retry 0: Received a response from request 42e35aff-c26b-4c24-9998-c965011536dc
azure:core-rest-pipeline retryPolicy:info Retry 0: Processing 1 retry strategies.
azure:core-rest-pipeline retryPolicy:info Retry 0: Processing retry strategy imdsRetryPolicy.
azure:core-rest-pipeline retryPolicy:info Retry 0: Skipped.
azure:core-rest-pipeline retryPolicy:info None of the retry strategies could work with the received response. Returning it.
azure:identity:error ManagedIdentityCredential => getToken() => ERROR. Scopes: https://vault.azure.net/.default. Error message: network_error: Network request failed.
azure:identity:info ChainedTokenCredential => getToken() => ERROR. Scopes: https://vault.azure.net/.default. Error message: ChainedTokenCredential authentication failed.
CredentialUnavailableError: ManagedIdentityCredential: Network unreachable. Message: network_error: Network request failed.
Error when setting up .env file: AggregateAuthenticationError: ChainedTokenCredential authentication failed.
CredentialUnavailableError: ManagedIdentityCredential: Network unreachable. Message: network_error: Network request failed
errors: [
CredentialUnavailableError: ManagedIdentityCredential: Network unreachable. Message: network_error: Network request failed
[cause]: [ClientAuthError]
}
]
}

Network Trace (Preferrably Fiddler)

• Sent
• Pending

MSAL Configuration

MSAL is wrapped by @azure/identity package.

Relevant Code Snippets

if (process.env.AZURE_CLIENT_ID) {
		credentialOptions.push(new ManagedIdentityCredential({ clientId: process.env.AZURE_CLIENT_ID }));
}

Reproduction Steps

1. Create Azure ML Compute Instance
2. Assign user assigned MI to Compute
3. Schedule job using @azure/identity package to use ManagedIdentityCredential

Expected Behavior

@azure/identity can pick up MI to fetch secrets from KV

Identity Provider

Entra ID (formerly Azure AD) / MSA

Browsers Affected (Select all that apply)

None (Server)

Regression

@azure/[email protected]