Started crushing

[sequencerr]

I guess it suddenly started failing to see libcuda, I fixed it by sudo modprobe nvidia_uvm, but imo the app shouldn't crash.

logs.txt

Couldn't find '/home/yrch/.ollama/id_ed25519'. Generating new private key.
2025/01/08 18:27:29 routes.go:1259: INFO server config env="map[CUDA_VISIBLE_DEVICES:GPU-afbafdd9-505a-55ff-00c0-1f07a0045ec7 GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY: HTTP_PROXY: NO_PROXY: OLLAMA_DEBUG:true OLLAMA_FLASH_ATTENTION:false OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://127.0.0.1:11435 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:5m0s OLLAMA_KV_CACHE_TYPE: OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:5m0s OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:/home/yrch/.var/app/com.jeffser.Alpaca/data/.ollama/models OLLAMA_MULTIUSER_CACHE:false OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:0 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://* vscode-webview://*] OLLAMA_SCHED_SPREAD:false ROCR_VISIBLE_DEVICES: http_proxy: https_proxy: no_proxy:]"
Your new public key is: 

ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKEYtImnVBPagXzfrPnT+yR9F/fMcS7savTIYjvjAq9k

time=2025-01-08T18:27:29.693+01:00 level=INFO source=images.go:757 msg="total blobs: 23"
[GIN-debug] [WARNING] Creating an Engine instance with the Logger and Recovery middleware already attached.

[GIN-debug] [WARNING] Running in "debug" mode. Switch to "release" mode in production.
 - using env:	export GIN_MODE=release
 - using code:	gin.SetMode(gin.ReleaseMode)

time=2025-01-08T18:27:29.693+01:00 level=INFO source=images.go:764 msg="total unused blobs removed: 0"
time=2025-01-08T18:27:29.693+01:00 level=INFO source=routes.go:1310 msg="Listening on 127.0.0.1:11435 (version 0.5.4)"
[GIN-debug] POST   /api/pull                 --> github.com/ollama/ollama/server.(*Server).PullHandler-fm (5 handlers)
[GIN-debug] POST   /api/generate             --> github.com/ollama/ollama/server.(*Server).GenerateHandler-fm (5 handlers)
[GIN-debug] POST   /api/chat                 --> github.com/ollama/ollama/server.(*Server).ChatHandler-fm (5 handlers)
[GIN-debug] POST   /api/embed                --> github.com/ollama/ollama/server.(*Server).EmbedHandler-fm (5 handlers)
[GIN-debug] POST   /api/embeddings           --> github.com/ollama/ollama/server.(*Server).EmbeddingsHandler-fm (5 handlers)
[GIN-debug] POST   /api/create               --> github.com/ollama/ollama/server.(*Server).CreateHandler-fm (5 handlers)
[GIN-debug] POST   /api/push                 --> github.com/ollama/ollama/server.(*Server).PushHandler-fm (5 handlers)
[GIN-debug] POST   /api/copy                 --> github.com/ollama/ollama/server.(*Server).CopyHandler-fm (5 handlers)
time=2025-01-08T18:27:29.693+01:00 level=DEBUG source=common.go:80 msg="runners located" dir=/app/lib/ollama/runners
[GIN-debug] DELETE /api/delete               --> github.com/ollama/ollama/server.(*Server).DeleteHandler-fm (5 handlers)
[GIN-debug] POST   /api/show                 --> github.com/ollama/ollama/server.(*Server).ShowHandler-fm (5 handlers)
[GIN-debug] POST   /api/blobs/:digest        --> github.com/ollama/ollama/server.(*Server).CreateBlobHandler-fm (5 handlers)
[GIN-debug] HEAD   /api/blobs/:digest        --> github.com/ollama/ollama/server.(*Server).HeadBlobHandler-fm (5 handlers)
[GIN-debug] GET    /api/ps                   --> github.com/ollama/ollama/server.(*Server).PsHandler-fm (5 handlers)
time=2025-01-08T18:27:29.693+01:00 level=DEBUG source=common.go:124 msg="availableServers : found" file=/app/lib/ollama/runners/cpu_avx/ollama_llama_server
time=2025-01-08T18:27:29.694+01:00 level=DEBUG source=common.go:124 msg="availableServers : found" file=/app/lib/ollama/runners/cpu_avx2/ollama_llama_server
time=2025-01-08T18:27:29.694+01:00 level=DEBUG source=common.go:124 msg="availableServers : found" file=/app/lib/ollama/runners/cuda_v11_avx/ollama_llama_server
time=2025-01-08T18:27:29.694+01:00 level=DEBUG source=common.go:124 msg="availableServers : found" file=/app/lib/ollama/runners/cuda_v12_avx/ollama_llama_server
time=2025-01-08T18:27:29.694+01:00 level=DEBUG source=common.go:124 msg="availableServers : found" file=/app/lib/ollama/runners/rocm_avx/ollama_llama_server
[GIN-debug] POST   /v1/chat/completions      --> github.com/ollama/ollama/server.(*Server).ChatHandler-fm (6 handlers)
[GIN-debug] POST   /v1/completions           --> github.com/ollama/ollama/server.(*Server).GenerateHandler-fm (6 handlers)
[GIN-debug] POST   /v1/embeddings            --> github.com/ollama/ollama/server.(*Server).EmbedHandler-fm (6 handlers)
time=2025-01-08T18:27:29.694+01:00 level=INFO source=routes.go:1339 msg="Dynamic LLM libraries" runners="[cpu cpu_avx cpu_avx2 cuda_v11_avx cuda_v12_avx rocm_avx]"
time=2025-01-08T18:27:29.694+01:00 level=DEBUG source=routes.go:1340 msg="Override detection logic by setting OLLAMA_LLM_LIBRARY"
time=2025-01-08T18:27:29.694+01:00 level=DEBUG source=sched.go:105 msg="starting llm scheduler"
time=2025-01-08T18:27:29.694+01:00 level=INFO source=gpu.go:226 msg="looking for compatible GPUs"
time=2025-01-08T18:27:29.694+01:00 level=DEBUG source=gpu.go:99 msg="searching for GPU discovery libraries for NVIDIA"
[GIN-debug] GET    /v1/models                --> github.com/ollama/ollama/server.(*Server).ListHandler-fm (6 handlers)
time=2025-01-08T18:27:29.694+01:00 level=DEBUG source=gpu.go:517 msg="Searching for GPU library" name=libcuda.so*
time=2025-01-08T18:27:29.695+01:00 level=DEBUG source=gpu.go:543 msg="gpu library search" globs="[/app/lib/ollama/libcuda.so* /app/lib/ollama/libcuda.so* /app/lib/libcuda.so* /usr/lib/x86_64-linux-gnu/GL/default/lib/libcuda.so* /usr/lib/x86_64-linux-gnu/openh264/extra/libcuda.so* /usr/lib/x86_64-linux-gnu/openh264/extra/libcuda.so* /usr/lib/sdk/llvm15/lib/libcuda.so* /usr/lib/x86_64-linux-gnu/GL/default/lib/libcuda.so* /usr/lib/ollama/libcuda.so* /app/plugins/AMD/lib/ollama/libcuda.so* /usr/local/cuda*/targets/*/lib/libcuda.so* /usr/lib/*-linux-gnu/nvidia/current/libcuda.so* /usr/lib/*-linux-gnu/libcuda.so* /usr/lib/wsl/lib/libcuda.so* /usr/lib/wsl/drivers/*/libcuda.so* /opt/cuda/lib*/libcuda.so* /usr/local/cuda/lib*/libcuda.so* /usr/lib*/libcuda.so* /usr/local/lib*/libcuda.so*]"
time=2025-01-08T18:27:29.696+01:00 level=DEBUG source=gpu.go:577 msg="discovered GPU libraries" paths=[]
time=2025-01-08T18:27:29.696+01:00 level=DEBUG source=gpu.go:517 msg="Searching for GPU library" name=libcudart.so*
time=2025-01-08T18:27:29.696+01:00 level=DEBUG source=gpu.go:543 msg="gpu library search" globs="[/app/lib/ollama/libcudart.so* /app/lib/ollama/libcudart.so* /app/lib/libcudart.so* /usr/lib/x86_64-linux-gnu/GL/default/lib/libcudart.so* /usr/lib/x86_64-linux-gnu/openh264/extra/libcudart.so* /usr/lib/x86_64-linux-gnu/openh264/extra/libcudart.so* /usr/lib/sdk/llvm15/lib/libcudart.so* /usr/lib/x86_64-linux-gnu/GL/default/lib/libcudart.so* /usr/lib/ollama/libcudart.so* /app/plugins/AMD/lib/ollama/libcudart.so* /app/lib/ollama/libcudart.so* /app/lib/ollama/libcudart.so* /usr/local/cuda/lib64/libcudart.so* /usr/lib/x86_64-linux-gnu/nvidia/current/libcudart.so* /usr/lib/x86_64-linux-gnu/libcudart.so* /usr/lib/wsl/lib/libcudart.so* /usr/lib/wsl/drivers/*/libcudart.so* /opt/cuda/lib64/libcudart.so* /usr/local/cuda*/targets/aarch64-linux/lib/libcudart.so* /usr/lib/aarch64-linux-gnu/nvidia/current/libcudart.so* /usr/lib/aarch64-linux-gnu/libcudart.so* /usr/local/cuda/lib*/libcudart.so* /usr/lib*/libcudart.so* /usr/local/lib*/libcudart.so*]"
[GIN-debug] GET    /v1/models/:model         --> github.com/ollama/ollama/server.(*Server).ShowHandler-fm (6 handlers)
[GIN-debug] GET    /                         --> github.com/ollama/ollama/server.(*Server).GenerateRoutes.func1 (5 handlers)
[GIN-debug] GET    /api/tags                 --> github.com/ollama/ollama/server.(*Server).ListHandler-fm (5 handlers)
[GIN-debug] GET    /api/version              --> github.com/ollama/ollama/server.(*Server).GenerateRoutes.func2 (5 handlers)
[GIN-debug] HEAD   /                         --> github.com/ollama/ollama/server.(*Server).GenerateRoutes.func1 (5 handlers)
[GIN-debug] HEAD   /api/tags                 --> github.com/ollama/ollama/server.(*Server).ListHandler-fm (5 handlers)
time=2025-01-08T18:27:29.696+01:00 level=DEBUG source=gpu.go:577 msg="discovered GPU libraries" paths="[/app/lib/ollama/libcudart.so.11.3.109 /app/lib/ollama/libcudart.so.12.4.127]"
[GIN-debug] HEAD   /api/version              --> github.com/ollama/ollama/server.(*Server).GenerateRoutes.func2 (5 handlers)
cudaSetDevice err: 999
time=2025-01-08T18:27:29.855+01:00 level=DEBUG source=gpu.go:593 msg="Unable to load cudart library /app/lib/ollama/libcudart.so.11.3.109: cudart init failure: 999"

[sequencerr]

before that I wanted to address https://github.com/ollama/ollama/blob/main/docs/gpu.md#laptop-suspend-resume issue, and #444 (comment), sudo rmmod nvidia_uvm && sudo modprobe nvidia_uvm didn't really fix. even if run 3b model, before it was using gpu normally.