Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug] 每隔一段时间服务器对所有涉及trpc的请求都无响应, 直到客户端超时 #6663

Open
sideef5ect opened this issue Mar 3, 2025 · 3 comments
Open

[Bug] 每隔一段时间服务器对所有涉及trpc的请求都无响应, 直到客户端超时 #6663

sideef5ect opened this issue Mar 3, 2025 · 3 comments
Labels
🐛 Bug Something isn't working | 缺陷

Comments

@sideef5ect
Copy link

sideef5ect commented Mar 3, 2025 •
edited
Loading

📦 Platform

Self hosting Docker

📦 Deploymenet mode

server db(lobe-chat-database image)

📌 Version

1.66.6

💻 Operating System

Other Linux, Ubuntu

🌐 Browser

Chrome

🐛 Bug Description

服务器运行大约30分钟后，TRPC请求持续失败并出现超时错误（HTTP 524）。服务器重启后会暂时恢复正常，但在类似时长后再次遇到相同问题。

如果在Chrome开发者工具中使用GET请求，服务器无响应，等待直到超时，然后返回524响应。

Image

All trpc requests failed with timeout 524 responses.

The server could work after restart, but only for a while like 30 mins, and then the trpc can not work any more.

📷 Recurrence Steps

TRPC requests to the server consistently fail with a timeout error (HTTP 524) after the server has been running for approximately 30 minutes. The server resumes normal operation temporarily after a restart but encounters the same issue again after a similar duration.

if use the GET request from chrome dev tools, the server will freeze, wait until timeout, and give a 524 response

curl -X GET "http://<PLACEHOLDER_DOMAIN>:3210/trpc/lambda/agent.getAgentConfig,aiProvider.getAiProviderRuntimeState,user.getUserState?batch=1&input=%7B%220%22%3A%7B%22json%22%3A%7B%22sessionId%22%3A%22inbox%22%7D%7D%2C%221%22%3A%7B%22json%22%3A%7B%22isLogin%22%3Anull%7D%2C%22meta%22%3A%7B%22values%22%3A%7B%22isLogin%22%3A%5B%22undefined%22%5D%7D%7D%7D%2C%222%22%3A%7B%22json%22%3Anull%2C%22meta%22%3A%7B%22values%22%3A%5B%22undefined%22%5D%7D%7D%7D" \
-H "Accept: */*" \
-H "Accept-Encoding: gzip, deflate, br, zstd" \
-H "Accept-Language: en,zh;q=0.9" \
-H "Content-Type: application/json" \
-H "Cookie: __Host-authjs.csrf-token=<PLACEHOLDER_CSRF_TOKEN>; LOBE_LOCALE=en-US; LOBE_THEME_APPEARANCE=dark; __Secure-authjs.callback-url=https%3A%2F%2F<PLACEHOLDER_DOMAIN>%2Fchat; ory_kratos_session=<PLACEHOLDER_SESSION>; __Secure-authjs.session-token=<PLACEHOLDER_SESSION_TOKEN>" \
-H "DNT: 1" \
-H "Priority: u=1, i" \
-H "Referer: https://<PLACEHOLDER_DOMAIN>/" \
-H "Sec-CH-UA: \"Chromium\";v=\"133\", \"Not(A:Brand\";v=\"99\"" \
-H "Sec-CH-UA-Mobile: ?0" \
-H "Sec-CH-UA-Platform: \"macOS\"" \
-H "Sec-Fetch-Dest: empty" \
-H "Sec-Fetch-Mode: cors" \
-H "Sec-Fetch-Site: same-origin" \
-H "User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/133.0.0.0 Safari/537.36" \
-H "X-Lobe-Chat-Auth: <PLACEHOLDER_AUTH_TOKEN>" -vvv
* Host lobe-chat:3210 was resolved.
* IPv6: (none)
* IPv4: 10.0.3.14, 10.0.3.20
*   Trying 10.0.3.14:3210...
* Connected to lobe-chat (10.0.3.14) port 3210
> GET /trpc/lambda/agent.getAgentConfig,aiProvider.getAiProviderRuntimeState,user.getUserState?batch=1&input=%7B%220%22%3A%7B%22json%22%3A%7B%22sessionId%22%3A%22inbox%22%7D%7D%2C%221%22%3A%7B%22json%22%3A%7B%22isLogin%22%3Anull%7D%2C%22meta%22%3A%7B%22values%22%3A%7B%22isLogin%22%3A%5B%22undefined%22%5D%7D%7D%7D%2C%222%22%3A%7B%22json%22%3Anull%2C%22meta%22%3A%7B%22values%22%3A%5B%22undefined%22%5D%7D%7D%7D HTTP/1.1
> Host: lobe-chat:3210
> Accept: */*
> Accept-Encoding: gzip, deflate, br, zstd
> Accept-Language: en,zh;q=0.9
> Content-Type: application/json
> Cookie: __Host-authjs.csrf-token=<PLACEHOLDER_CSRF_TOKEN>; LOBE_LOCALE=en-US; LOBE_THEME_APPEARANCE=dark; __Secure-authjs.callback-url=https%3A%2F%2F<PLACEHOLDER_DOMAIN>%2Fchat; ory_kratos_session=<PLACEHOLDER_SESSION>; __Secure-authjs.session-token=<PLACEHOLDER_SESSION_TOKEN>
> DNT: 1
> Priority: u=1, i
> Referer: https://<PLACEHOLDER_DOMAIN>/
> Sec-CH-UA: "Chromium";v="133", "Not(A:Brand";v="99"
> Sec-CH-UA-Mobile: ?0
> Sec-CH-UA-Platform: "macOS"
> Sec-Fetch-Dest: empty
> Sec-Fetch-Mode: cors
> Sec-Fetch-Site: same-origin
> User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/133.0.0.0 Safari/537.36
> X-Lobe-Chat-Auth: <PLACEHOLDER_AUTH_TOKEN>
>

if use POST, it server could immediately return error.

curl -X POST "http://lobe-chat:3210/trpc/lambda/agent.getAgentConfig,aiProvider.getAiProviderRuntimeState,user.getUserState"
{"error":{"json":{"message":"Missing content-type header","code":-32015,"data":{"code":"UNSUPPORTED_MEDIA_TYPE","httpStatus":415}}}}

🚦 Expected Behavior

no error, or return error response immediately

📝 Additional Information

api gateway: nginx
upgrade to latest lobe-chat-database images seems still suffer from the same issue
@sideef5ect sideef5ect added the 🐛 Bug Something isn't working | 缺陷 label Mar 3, 2025
@github-project-automation github-project-automation bot moved this to Roadmap - Chat 1.x in Lobe Chat Routine Mar 3, 2025
@lobehubbot
Copy link
Member

👀 @sideef5ect

Thank you for raising an issue. We will investigate into the matter and get back to you as soon as possible.
Please make sure you have given us as much context as possible.
非常感谢您提交 issue。我们会尽快调查此事，并尽快回复您。 请确保您已经提供了尽可能多的背景信息。

@sideef5ect sideef5ect changed the title [Bug] tRPC server freeze [Bug] All TRPC Requests Timeout Mar 3, 2025
@sideef5ect
Copy link
Author

Anyone could help?🥺

@sideef5ect sideef5ect changed the title [Bug] All TRPC Requests Timeout [Bug] 每隔一段时间服务器对所有涉及trpc的请求都无响应, 直到客户端超时 Mar 4, 2025
@sideef5ect
Copy link
Author

server logs

docker service logs lobe-chat
lobe-chat.1.tzfxhfqgspbo@node2    | 🌐 DNS Server: [ '127.0.0.11' ]
lobe-chat.1.tzfxhfqgspbo@node2    | -------------------------------------
lobe-chat.1.tzfxhfqgspbo@node2    | [Database] Start to migration...
lobe-chat.1.tzfxhfqgspbo@node2    | ✅ database migration pass.
lobe-chat.1.tzfxhfqgspbo@node2    | -------------------------------------
lobe-chat.1.tzfxhfqgspbo@node2    |    ▲ Next.js 15.2.0
lobe-chat.1.tzfxhfqgspbo@node2    |    - Local:        http://localhost:3210
lobe-chat.1.tzfxhfqgspbo@node2    |    - Network:      http://0.0.0.0:3210
lobe-chat.1.tzfxhfqgspbo@node2    |
lobe-chat.1.tzfxhfqgspbo@node2    |  ✓ Starting...
lobe-chat.1.tzfxhfqgspbo@node2    |  ✓ Ready in 39ms
lobe-chat.1.tzfxhfqgspbo@node2    | {
lobe-chat.1.tzfxhfqgspbo@node2    |   allowDangerousEmailAccountLinking: true,
lobe-chat.1.tzfxhfqgspbo@node2    |   clientId: undefined,
lobe-chat.1.tzfxhfqgspbo@node2    |   clientSecret: undefined,
lobe-chat.1.tzfxhfqgspbo@node2    |   platformType: 'WebsiteApp',
lobe-chat.1.tzfxhfqgspbo@node2    |   profile: [Function: profile]
lobe-chat.1.tzfxhfqgspbo@node2    | }
lobe-chat.1.tzfxhfqgspbo@node2    | {
lobe-chat.1.tzfxhfqgspbo@node2    |   allowDangerousEmailAccountLinking: true,
lobe-chat.1.tzfxhfqgspbo@node2    |   clientId: undefined,
lobe-chat.1.tzfxhfqgspbo@node2    |   clientSecret: undefined,
lobe-chat.1.tzfxhfqgspbo@node2    |   platformType: 'WebsiteApp',
lobe-chat.1.tzfxhfqgspbo@node2    |   profile: [Function: profile]
lobe-chat.1.tzfxhfqgspbo@node2    | }
lobe-chat.1.tzfxhfqgspbo@node2    | [rewrite] /chat -> http://127.0.0.1:3210/en-US__0__dark/chat
lobe-chat.1.tzfxhfqgspbo@node2    | (node:28) [DEP0060] DeprecationWarning: The `util._extend` API is deprecated. Please use Object.assign() instead.
lobe-chat.1.tzfxhfqgspbo@node2    | (Use `node --trace-deprecation ...` to show where the warning was created)
lobe-chat.1.tzfxhfqgspbo@node2    | [rewrite] /files -> http://127.0.0.1:3210/en-US__0__dark/files
lobe-chat.1.tzfxhfqgspbo@node2    | [rewrite] /chat -> http://127.0.0.1:3210/en-US__0__dark/chat
lobe-chat.1.tzfxhfqgspbo@node2    | [rewrite] / -> http://127.0.0.1:3210/en-US__0__dark
lobe-chat.1.tzfxhfqgspbo@node2    | {
lobe-chat.1.tzfxhfqgspbo@node2    |   allowDangerousEmailAccountLinking: true,
lobe-chat.1.tzfxhfqgspbo@node2    |   clientId: undefined,
lobe-chat.1.tzfxhfqgspbo@node2    |   clientSecret: undefined,
lobe-chat.1.tzfxhfqgspbo@node2    |   platformType: 'WebsiteApp',
lobe-chat.1.tzfxhfqgspbo@node2    |   profile: [Function: profile]
lobe-chat.1.tzfxhfqgspbo@node2    | }
lobe-chat.1.tzfxhfqgspbo@node2    | [rewrite] / -> http://127.0.0.1:3210/en-US__0__dark
lobe-chat.1.tzfxhfqgspbo@node2    | [rewrite] /discover -> http://127.0.0.1:3210/en-US__0__dark/discover
lobe-chat.1.tzfxhfqgspbo@node2    | [rewrite] /chat -> http://127.0.0.1:3210/en-US__0__dark/chat
lobe-chat.1.tzfxhfqgspbo@node2    | [rewrite] /chat -> http://127.0.0.1:3210/en-US__0__dark/chat
lobe-chat.1.tzfxhfqgspbo@node2    | [rewrite] / -> http://127.0.0.1:3210/en-US__0__dark
lobe-chat.1.tzfxhfqgspbo@node2    | [rewrite] /chat -> http://127.0.0.1:3210/en-US__0__dark/chat
lobe-chat.1.tzfxhfqgspbo@node2    | [rewrite] /files -> http://127.0.0.1:3210/en-US__0__dark/files
lobe-chat.1.tzfxhfqgspbo@node2    | [rewrite] /discover -> http://127.0.0.1:3210/en-US__0__dark/discover
lobe-chat.1.tzfxhfqgspbo@node2    | [rewrite] /discover/assistant/bad-language-helper -> http://127.0.0.1:3210/en-US__0__dark/discover/assistant/bad-language-helper
lobe-chat.1.tzfxhfqgspbo@node2    | [rewrite] /discover/assistant/crontab-generate -> http://127.0.0.1:3210/en-US__0__dark/discover/assistant/crontab-generate
lobe-chat.1.tzfxhfqgspbo@node2    | [rewrite] /discover/assistant/fate-researcher -> http://127.0.0.1:3210/en-US__0__dark/discover/assistant/fate-researcher
lobe-chat.1.tzfxhfqgspbo@node2    | [rewrite] /discover/assistant/xiao-zhi-french-translation-asst-v-1 -> http://127.0.0.1:3210/en-US__0__dark/discover/assistant/xiao-zhi-french-translation-asst-v-1
lobe-chat.1.tzfxhfqgspbo@node2    | [rewrite] /chat -> http://127.0.0.1:3210/en-US__0__dark/chat
lobe-chat.1.tzfxhfqgspbo@node2    | [rewrite] /discover -> http://127.0.0.1:3210/en-US__0__dark/discover
lobe-chat.1.tzfxhfqgspbo@node2    | [rewrite] /files -> http://127.0.0.1:3210/en-US__0__dark/files
lobe-chat.1.tzfxhfqgspbo@node2    | [rewrite] /chat -> http://127.0.0.1:3210/en-US__0__dark/chat
lobe-chat.1.tzfxhfqgspbo@node2    | [Error: aborted] { code: 'ECONNRESET' }
lobe-chat.1.tzfxhfqgspbo@node2    |  ⨯ uncaughtException:  [Error: aborted] { code: 'ECONNRESET' }
lobe-chat.1.tzfxhfqgspbo@node2    | [rewrite] /files -> http://127.0.0.1:3210/en-US__0__dark/files
lobe-chat.1.tzfxhfqgspbo@node2    | [rewrite] /discover -> http://127.0.0.1:3210/en-US__0__dark/discover
lobe-chat.1.tzfxhfqgspbo@node2    | [rewrite] / -> http://127.0.0.1:3210/en-US__0__dark

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
🐛 Bug Something isn't working | 缺陷
Projects
Lobe Chat Routine
Status: Roadmap - Chat 1.x
Milestone
No milestone
Development

No branches or pull requests
2 participants
@sideef5ect @lobehubbot