Deploying infinity in a docker, stucks at loading the model.

[ouz-m]

Hello,

I am trying to deploy infinity with the model erst/xlm-roberta-base-finetuned-nace, when i use the official docker image with given commands it stucks at this stage.

port=7997
model1=erst/xlm-roberta-base-finetuned-nace
volume=$PWD/data

docker run \
 -v $volume:/app/.cache \
 -p $port:$port \
 michaelf34/infinity:latest \
 v2 \
 --model-id $model1 \
 --port $port
INFO:     Started server process [1]
INFO:     Waiting for application startup.
INFO     2024-06-06 13:34:30,011 infinity_emb INFO:           select_model.py:54
         model=`erst/xlm-roberta-base-finetuned-nace`                           
         selected, using engine=`torch` and device=`None`                       

deployment in my cli for the same model works perfectly fine, also other models such as michaelfeil/bge-small-en-v1.5 works with no problem.

then i decided to create a simple custom container and try it out,

FROM python:3.10 as base

ENV PYTHONUNBUFFERED=1 \
    \
    # pip
    PIP_NO_CACHE_DIR=off \
    PIP_DISABLE_PIP_VERSION_CHECK=on \
    PIP_DEFAULT_TIMEOUT=100 \
    \
    PYTHON="python3.10"

RUN apt-get update && apt-get install -y \
    build-essential \
    python3-dev \
    python3-venv \
    curl

WORKDIR /app/


ENV VIRTUAL_ENV=/opt/venv
RUN python3 -m venv $VIRTUAL_ENV
ENV PATH="$VIRTUAL_ENV/bin:$PATH"

# Install dependencies:
COPY requirements.txt .
RUN pip install -r requirements.txt

ENV HF_HOME=/app/.cache/huggingface

# Use a multi-stage build -> production version
FROM base AS production
ENTRYPOINT ["infinity_emb"]

note: I am newbie at containerisation :P

still have the same problem.

I would appreciate if anyone can help me with it, thanks!!

[michaelfeil]

I think the model might be downloading? Or running out of cuda/memory?

[ouz-m]

hi @michaelfeil, I think you are correct, after I increased the memory it started without any problems. Thanks!!