Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: Cannot reliably interface with local LLM when agent is built from source #258

Open
kevin-support-bot bot opened this issue Feb 7, 2025 · 25 comments
Open

[Bug]: Cannot reliably interface with local LLM when agent is built from source #258

kevin-support-bot bot opened this issue Feb 7, 2025 · 25 comments

Comments

@kevin-support-bot
Copy link

All-Hands-AI#6643 Issue

@avi12, Does `curl base_url work in WSL terminal?
@avi12
Copy link

avi12 commented Feb 7, 2025

Do you mean to run:

curl base_url

@SmartManoj
Copy link
Owner

curl http://127.0.0.1:1234/v1

@avi12
Copy link

avi12 commented Feb 7, 2025

huh

avi12@avi12:~/OpenHands$ curl 127.0.0.1:1234/v1
curl: (7) Failed to connect to 127.0.0.1 port 1234 after 0 ms: Connection refused
avi12@avi12:~/OpenHands$ curl http://127.0.0.1:1234/v1
curl: (7) Failed to connect to 127.0.0.1 port 1234 after 0 ms: Connection refused
avi12@avi12:~/OpenHands$ curl http://host.internal.docker:1234/v1
curl: (6) Could not resolve host: host.internal.docker

@SmartManoj
Copy link
Owner

Run the following commands in cmd to set up the networking mode of WSL to mirrored:

python -c  "print('[wsl2]\nnetworkingMode=mirrored',file=open(r'%UserProfile%\.wslconfig','w'))"
wsl --shutdown

@avi12
Copy link

avi12 commented Feb 7, 2025

It seems like it worked and after the shutdown, Docker Desktop asked me to restart WSL, I hit restart and then I could do

curl host.docker.internal:1234/v1

and it didn't return an error

@SmartManoj
Copy link
Owner

curl http://127.0.0.1:1234/v1 works?

@avi12
Copy link

avi12 commented Feb 7, 2025

No, only host.docker.internal:1234/v1 works

@avi12
Copy link

avi12 commented Feb 7, 2025

I tried the following combinations for which model to specify:

  1. model="lm_studio/qwen2.5-coder-7b-instruct"
  2. model="lmstudio-community/qwen2.5-coder-7b-instruct"
  3. model="huggingface/qwen2.5-coder-7b-instruct"
  4. model="qwen2.5-coder-7b-instruct"

2 and 4 trigger an error, saying that it cannot interface with the model
1 and 3 don't trigger LM Studio

@SmartManoj
Copy link
Owner

https://docs.litellm.ai/docs/providers/lm_studio#sample-usage

lm_studio/{model_name}

host.docker.internal:1234/v1/models to see the model name.

@avi12
Copy link

avi12 commented Feb 7, 2025

I'm so confused
When I try to do curl host.docker.internal:1234/v1/models, it hangs
but Postman works with http://127.0.0.1:1234/v1/models

@SmartManoj
Copy link
Owner

In cmd, python -m http.server
And in WSL, does curl http://localhost:8000 work?

@avi12
Copy link

avi12 commented Feb 7, 2025 

avi12@avi12:~/OpenHands$ curl localhost:8000
curl: (7) Failed to connect to localhost port 8000 after 0 ms: Connection refused

@SmartManoj
Copy link
Owner

microsoft/WSL#10791

@avi12
Copy link

avi12 commented Feb 7, 2025

What do I do?

@avi12 avi12 mentioned this issue Feb 7, 2025
Open
1 task
@SmartManoj
Copy link
Owner

curl to the local IP address of windows works in WSL?

@avi12
Copy link

avi12 commented Feb 8, 2025

it doesn't

@SmartManoj
Copy link
Owner

Did you check this?

@avi12
Copy link

avi12 commented Feb 8, 2025

I did, I assume I have to run in the WSL terminal

sudo systemctl mask systemd-networkd

which I did, but even restarting the terminal results in

avi12@avi12:~/OpenHands$ curl 127.0.0.1:8000
curl: (7) Failed to connect to 127.0.0.1 port 8000 after 0 ms: Connection refused

@SmartManoj
Copy link
Owner

It should be discussed in the WSL repo.

microsoft/WSL#10650 (comment)

@SmartManoj
Copy link
Owner

SmartManoj commented Feb 8, 2025 •
edited
Loading

Does curl google.com work? If so, using ngrok would be a workaround.

@avi12
Copy link

avi12 commented Feb 8, 2025

in .wslconfig I set

[wsl2]
networkingMode=mirrored
hostAddressLoopback=true

then from PowerShell I did

wsl --shutdown

then I tried to curl 127.0.0.1:8000, still failed

@avi12
Copy link

avi12 commented Feb 8, 2025

Does curl google.com work?

yep

<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>301 Moved</TITLE></HEAD><BODY>
<H1>301 Moved</H1>
The document has moved
<A HREF="http://www.google.com/">here</A>.
</BODY></HTML>

@avi12
Copy link

avi12 commented Feb 8, 2025

After digging into the WSL config docs, it seems like
Image
can only be enabled on Windows 11 😐

I guess so long I'm on Windows 10 due to my old hardware, my only option is to use services like Ngrok

@SmartManoj
Copy link
Owner

curl to the local IP address of Windows works in WSL?

But this works in Windows 10.

@avi12
Copy link

avi12 commented Feb 9, 2025

I couldn't manage to get it working, unless it can work in WSL1, certainly not in WSL2

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@avi12 @SmartManoj