bug: Unable to run script as /bin/sh: 1: cannot open 4.0: No such file

[lucashicks1]

Describe the bug

Hi,
I am just getting into using windmill and am encountering an issue when trying to run my script. It is a simple python script that uses the Azure python SDK to check if a resource group exists and if it doesn't, it creates one.

I am running this through the web ui on my locally hosted instance of windmill (just using docker compose up) since Run Preview in VScode isn't working as it doesn't seem to pick up a requirements.txt file (this file is needed because windmill keeps trying to run pip install azure instead of pip install azure-mgmt-... which causes an error because of this).

The error that is appearing is:

ExecutionErr: ExitCode: 2, last log lines:
nstalling collected packages: anyio
Successfully installed anyio-4.4.0
WARNING: Target directory /tmp/windmill/cache/pip/anyio==4.4.0/anyio already exists. Specify --upgrade to force replacement.
WARNING: Target directory /tmp/windmill/cache/pip/anyio==4.4.0/anyio-4.4.0.dist-info already exists. Specify --upgrade to force replacement.
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
/bin/sh: 1: cannot open 4.0: No such file

To reproduce

1. Develop the script locally in VSCode
2. Call wmill script generate-metadata to generate the lock file
3. Call wmill sync push to push my script to the instance
4. Run the script using the web UI then get the error

Expected behavior

Above steps just without the error.

Screenshots

No response

Browser information

MS Edge Version 128.0.2739.67

Application version

v1.394.2-2-g562b2fc67

Additional Context

No response

[rubenfiszel]

Is the lockfile generated correct and is it correctly pushed for that script (you can see it in the details tab of that script in the ui)

[lucashicks1]

Yes I think so, I've run the following just to double check.

user: windmill-flows$ wmill script generate-metadata
Generating metadata for all stale scripts:
Found raw requirements (package.json/requirements.txt/composer.json) for f/azure/resource_groups/scripts/create_rg_if_not_exists.py, using it
No metadata to update
user: windmill-flows$ wmill sync push
Computing the files to update on the remote to match local (taking wmill.yaml includes/excludes into account)
Remote version: CE v1.394.2-2-g562b2fc67
remote (main) <- local: 0 changes to apply
user: windmill-flows$

I have also run it again and am still getting the same issue.

[rubenfiszel]

Can you share that lockfile?

[lucashicks1]

Sure thing

anyio==4.4.0 ; python_version >= "3.11" and python_version < "4.0"
azure-common==1.1.28 ; python_version >= "3.11" and python_version < "4.0"
azure-core==1.30.2 ; python_version >= "3.11" and python_version < "4.0"
azure-identity==1.17.1 ; python_version >= "3.11" and python_version < "4.0"
azure-mgmt-core==1.4.0 ; python_version >= "3.11" and python_version < "4.0"
azure-mgmt-resource==23.1.1 ; python_version >= "3.11" and python_version < "4.0"
certifi==2024.8.30 ; python_version >= "3.11" and python_version < "4.0"
cffi==1.17.1 ; python_version >= "3.11" and python_version < "4.0" and platform_python_implementation != "PyPy"
charset-normalizer==3.3.2 ; python_version >= "3.11" and python_version < "4.0"
cryptography==43.0.1 ; python_version >= "3.11" and python_version < "4.0"
h11==0.14.0 ; python_version >= "3.11" and python_version < "4.0"
httpcore==1.0.5 ; python_version >= "3.11" and python_version < "4.0"
httpx==0.27.2 ; python_version >= "3.11" and python_version < "4.0"
idna==3.8 ; python_version >= "3.11" and python_version < "4.0"
isodate==0.6.1 ; python_version >= "3.11" and python_version < "4.0"
loguru==0.7.2 ; python_version >= "3.11" and python_version < "4.0"
msal==1.31.0 ; python_version >= "3.11" and python_version < "4.0"
msal-extensions==1.2.0 ; python_version >= "3.11" and python_version < "4.0"
portalocker==2.10.1 ; python_version >= "3.11" and python_version < "4.0"
pycparser==2.22 ; python_version >= "3.11" and python_version < "4.0" and platform_python_implementation != "PyPy"
pyjwt==2.9.0 ; python_version >= "3.11" and python_version < "4.0"
requests==2.32.3 ; python_version >= "3.11" and python_version < "4.0"
six==1.16.0 ; python_version >= "3.11" and python_version < "4.0"
sniffio==1.3.1 ; python_version >= "3.11" and python_version < "4.0"
typing-extensions==4.12.2 ; python_version >= "3.11" and python_version < "4.0"
urllib3==2.2.2 ; python_version >= "3.11" and python_version < "4.0"
wmill==1.394.3 ; python_version >= "3.11" and python_version < "4.0"

[rubenfiszel]

We don't support this format, remove everything after ;

anyio==4.4.0 
azure-common==1.1.28 
azure-core==1.30.2 
azure-identity==1.17.1 
azure-mgmt-core==1.4.0 
azure-mgmt-resource==23.1.1 
certifi==2024.8.30 
cffi==1.17.1 
charset-normalizer==3.3.2 
cryptography==43.0.1 
h11==0.14.0 
httpcore==1.0.5 
httpx==0.27.2 
idna==3.8 
isodate==0.6.1 
loguru==0.7.2 
msal==1.31.0 
msal-extensions==1.2.0 
portalocker==2.10.1 
pycparser==2.22 
pyjwt==2.9.0 
requests==2.32.3 
six==1.16.0 
sniffio==1.3.1 
typing-extensions==4.12.2 
urllib3==2.2.2 
wmill==1.394.3 

it will work

[lucashicks1]

With this change now, I'm unable to even generate the metadata for the script and am getting the following error:

user: flows-v2$ wmill script generate-metadata
Generating metadata for all stale scripts:
Found raw requirements (package.json/requirements.txt/composer.json) for f/azure/resource_groups/scripts/create_rg_if_not_exists.py, using it
+ f/azure/resource_groups/scripts/create_rg_if_not_exists (python3)
? Update the metadata of the above scripts? (Y/n) › Yes
Found raw requirements (package.json/requirements.txt/composer.json) for f/azure/resource_groups/scripts/create_rg_if_not_exists.py, using it
Generating metadata for f/azure/resource_groups/scripts/create_rg_if_not_exists.py
file:///usr/local/lib/node_modules/windmill-cli/esm/metadata.js:223
        throw new Error(`Failed to generate lockfile. Status was: ${rawResponse.statusText}, ${responseText}, ${e}`);
              ^

Error: Failed to generate lockfile. Status was: OK, {"error": {"name": "ExecutionErr", "message": "error during execution of the script:\nError locking file: Lock file generation failed: ExitStatus(1)"}}, Error: Failed to generate lockfile. Full response was: {"error":{"name":"ExecutionErr","message":"error during execution of the script:\nError locking file: Lock file generation failed: ExitStatus(1)"}}
    at updateScriptLock (file:///usr/local/lib/node_modules/windmill-cli/esm/metadata.js:223:15)
    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
    at async generateScriptMetadataInternal (file:///usr/local/lib/node_modules/windmill-cli/esm/metadata.js:136:13)
    at async Command.generateMetadata [as actionHandler] (file:///usr/local/lib/node_modules/windmill-cli/esm/script.js:636:13)
    at async Command.execute (file:///usr/local/lib/node_modules/windmill-cli/esm/deps/jsr.io/@windmill-labs/cliffy-command/1.0.0-rc.5/command.js:970:13)
    at async Command.parseCommand (file:///usr/local/lib/node_modules/windmill-cli/esm/deps/jsr.io/@windmill-labs/cliffy-command/1.0.0-rc.5/command.js:852:20)
    at async main (file:///usr/local/lib/node_modules/windmill-cli/esm/main.js:120:9)

Node.js v22.8.0

I've tried to reinstall the windmill cli, create a new copy of the workspace but still am getting this error. Without the requirements.txt file I get this error, but if I comment out all of the dependencies it disappears.

[lucashicks1]

Also out of curiosity, is the dependency inference for packages like the azure-sdk where their 'meta-package' is deprecated something that is easily fixable? Having imports like azure.mgmt.resource be resolve to azure-mgmt-resource instead of just azure would mean I could remove the requirements.txt file.

Being able to remove this file would be great as it would make things simpler but would more importantly allow scripts/flows to be previewed locally as currently, the VSCode extension doesn't pick up on the requirements.txt file.

[rubenfiszel]

I can't reproduce your issue with a requirements.txt with the content above:

anyio==4.4.0 
azure-common==1.1.28 
azure-core==1.30.2 
azure-identity==1.17.1 
azure-mgmt-core==1.4.0 
azure-mgmt-resource==23.1.1 
certifi==2024.8.30 
cffi==1.17.1 
charset-normalizer==3.3.2 
cryptography==43.0.1 
h11==0.14.0 
httpcore==1.0.5 
httpx==0.27.2 
idna==3.8 
isodate==0.6.1 
loguru==0.7.2 
msal==1.31.0 
msal-extensions==1.2.0 
portalocker==2.10.1 
pycparser==2.22 
pyjwt==2.9.0 
requests==2.32.3 
six==1.16.0 
sniffio==1.3.1 
typing-extensions==4.12.2 
urllib3==2.2.2 
wmill==1.394.3 

The VSCode extension can pick-up on the lockfile directly but you can also override the dependencies inference directly with:
https://www.windmill.dev/docs/advanced/dependencies_in_python#pinning-dependencies-and-requirements

[lucashicks1]

I'm also now getting this issue with typescript (deno), trying to generate a lock file from my package.json. Going through the error, it seems to be caught on line 380 (referencing line numbers on Github) in metadata.ts, as response.lock is undefined on line 358. From this, it seems that the POST request to ${workspace.remote}api/w/${workspace.workspaceId}/jobs/run/dependencies is failing.

It is sending the following as the body:

{
  raw_scripts: [
    {
      raw_code: '/// <reference types="node" />\n' +
        '/// <reference types="node" />\n' +
        '/// <reference types="node" />\n' +
        '/// <reference types="node" />\n' +
        "import * as net from 'net';\n" +
        "import * as tls from 'tls';\n" +
        "import * as http from 'http';\n" +
        "import { Agent, AgentConnectOpts } from 'agent-base';\n" +
        "import { URL } from 'url';\n" +
        "import type { OutgoingHttpHeaders } from 'http';\n" +
        'type Protocol<T> = T extends `${infer Protocol}:${infer _}` ? Protocol : never;\n' +
        'type ConnectOptsMap = {\n' +
        "    http: Omit<net.TcpNetConnectOpts, 'host' | 'port'>;\n" +
        "    https: Omit<tls.ConnectionOptions, 'host' | 'port'>;\n" +
        '};\n' +
        'type ConnectOpts<T> = {\n' +
        '    [P in keyof ConnectOptsMap]: Protocol<T> extends P ? ConnectOptsMap[P] : never;\n' +
        '}[keyof ConnectOptsMap];\n' +
        'export type HttpProxyAgentOptions<T> = ConnectOpts<T> & http.AgentOptions & {\n' +
        '    headers?: OutgoingHttpHeaders | (() => OutgoingHttpHeaders);\n' +
        '};\n' +
        'interface HttpProxyAgentClientRequest extends http.ClientRequest {\n' +
        '    outputData?: {\n' +
        '        data: string;\n' +
        '    }[];\n' +
        '    _header?: string | null;\n' +
        '    _implicitHeader(): void;\n' +
        '}\n' +
        '/**\n' +
        ' * The `HttpProxyAgent` implements an HTTP Agent subclass that connects\n' +
        ' * to the specified "HTTP proxy server" in order to proxy HTTP requests.\n' +
        ' */\n' +
        'export declare class HttpProxyAgent<Uri extends string> extends Agent {\n' +
        '    static protocols: readonly ["http", "https"];\n' +
        '    readonly proxy: URL;\n' +
        '    proxyHeaders: OutgoingHttpHeaders | (() => OutgoingHttpHeaders);\n' +
        '    connectOpts: net.TcpNetConnectOpts & tls.ConnectionOptions;\n' +
        '    constructor(proxy: Uri | URL, opts?: HttpProxyAgentOptions<Uri>);\n' +
        '    addRequest(req: HttpProxyAgentClientRequest, opts: AgentConnectOpts): void;\n' +
        '    setRequestProps(req: HttpProxyAgentClientRequest, opts: AgentConnectOpts): void;\n' +
        '    connect(req: HttpProxyAgentClientRequest, opts: AgentConnectOpts): Promise<net.Socket>;\n' +
        '}\n' +
        'export {};\n' +
        '//# sourceMappingURL=index.d.ts.map',
      language: 'bun',
      script_path: 'f/azure/common/node_modules/http-proxy-agent/dist/index'
    }
  ],
  raw_deps: '{\n' +
    '  "name": "http-proxy-agent",\n' +
    '  "version": "7.0.2",\n' +
    '  "description": "An HTTP(s) proxy `http.Agent` implementation for HTTP",\n' +
    '  "main": "./dist/index.js",\n' +
    '  "types": "./dist/index.d.ts",\n' +
    '  "files": [\n' +
    '    "dist"\n' +
    '  ],\n' +
    '  "repository": {\n' +
    '    "type": "git",\n' +
    '    "url": "https://github.com/TooTallNate/proxy-agents.git",\n' +
    '    "directory": "packages/http-proxy-agent"\n' +
    '  },\n' +
    '  "keywords": [\n' +
    '    "http",\n' +
    '    "proxy",\n' +
    '    "endpoint",\n' +
    '    "agent"\n' +
    '  ],\n' +
    '  "author": "Nathan Rajlich <[email protected]> (http://n8.io/)",\n' +
    '  "license": "MIT",\n' +
    '  "dependencies": {\n' +
    '    "agent-base": "^7.1.0",\n' +
    '    "debug": "^4.3.4"\n' +
    '  },\n' +
    '  "devDependencies": {\n' +
    '    "@types/debug": "^4.1.7",\n' +
    '    "@types/jest": "^29.5.1",\n' +
    '    "@types/node": "^14.18.45",\n' +
    '    "async-listen": "^3.0.0",\n' +
    '    "jest": "^29.5.0",\n' +
    '    "ts-jest": "^29.1.0",\n' +
    '    "typescript": "^5.0.4",\n' +
    '    "proxy": "2.1.1",\n' +
    '    "tsconfig": "0.0.0"\n' +
    '  },\n' +
    '  "engines": {\n' +
    '    "node": ">= 14"\n' +
    '  },\n' +
    '  "scripts": {\n' +
    '    "build": "tsc",\n' +
    '    "test": "jest --env node --verbose --bail",\n' +
    '    "lint": "eslint . --ext .ts",\n' +
    '    "pack": "node ../../scripts/pack.mjs"\n' +
    '  }\n' +
    '}',
  entrypoint: 'f/azure/common/node_modules/http-proxy-agent/dist/index'
}

and is getting the following response:

{
    "error": {
        "name": "ExecutionErr",
        "message": "error during execution of the script:\nError locking file: Non-zero exit status: 1"
    }
}

Is this error repeatable on your end?

[lucashicks1]

After, playing around just a bit more (I found it odd that the language field in the request body was set to bun when my script file was xyz.deno.ts). After looking into generateScriptMetadataInternal() and inferContentTypeFromFilePath() it seems like it was inferring the file type from f/azure/common/node_modules/http-proxy-agent/dist/index.d.ts instead of my actual script file. Once I deleted the node_modules folder, this issue was resolved <- and I guess I don't need these because of deno :)