Support for multilingual requests broken on openai==1.55.3 and httpx==0.28.0

[ishaan-jaff]

Confirm this is an issue with the Python library and not an underlying OpenAI API

• This is an issue with the Python library

Describe the bug

Support for multilingual requests broken on openai==1.55.3 and httpx==0.28.0

We use openai for multi lingual requests that contain multilingual chars, this used to work on previous openai versions

as soon as we upgraded openai to 1.55.3 and httpx==0.28.0 we got the following error

To Reproduce

use openai==1.55.3 and httpx==0.28.0 .

Run the request

you'll see this error

(The error goes away when we use httpx==0.27.0 but it would be nice if openai python could handle this for us)

E       UnicodeEncodeError: 'utf-8' codec can't encode character '\ud83e' in position 39: surrogates not allowed

/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/httpx/_content.py:179: UnicodeEncodeError

Code snippets

import openai
    client = openai.OpenAI(api_key=os.environ.get("OPENAI_API_KEY"))
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": "你好世界！\ud83e, ö"}],
    )

Traceback

test_openai.py:279:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/openai/_utils/_utils.py:275: in wrapper
    return func(*args, **kwargs)
/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/openai/resources/chat/completions.py:829: in create
    return self._post(
/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/openai/_base_client.py:1280: in post
    return cast(ResponseT, self.request(cast_to, opts, stream=stream, stream_cls=stream_cls))
/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/openai/_base_client.py:957: in request
    return self._request(
/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/openai/_base_client.py:983: in _request
    request = self._build_request(options, retries_taken=retries_taken)
/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/openai/_base_client.py:506: in _build_request
    return self._client.build_request(  # pyright: ignore[reportUnknownMemberType]
/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/httpx/_client.py:378: in build_request
    return Request(
/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/httpx/_models.py:408: in __init__
    headers, stream = encode_request(
/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/httpx/_content.py:216: in encode_request
    return encode_json(json)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

json = {'messages': [{'content': '\ud83e', 'role': 'user'}], 'model': 'gpt-4o-mini'}

    def encode_json(json: Any) -> tuple[dict[str, str], ByteStream]:
        body = json_dumps(
            json, ensure_ascii=False, separators=(",", ":"), allow_nan=False
>       ).encode("utf-8")
E       UnicodeEncodeError: 'utf-8' codec can't encode character '\ud83e' in position 39: surrogates not allowed

/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/httpx/_content.py:179: UnicodeEncodeError

OS

macOS

Python version

3.10

Library version

openai v0.27.0

[RobertCraigie]

Thanks for the report @ishaan-jaff, this sounds to me like an issue with httpx itself, can you reproduce this just using httpx directly?

[ishaan-jaff]

yes, it is an issue with httpx (not necessarily with openai python)

Is it possible that OpenAI python can add handling for this ?

Since this used to work before we upgraded to openai==1.55.3 and httpx==0.28.0

[RobertCraigie]

I'm not sure if we can. Would you be able to open an issue with httpx (if you haven't already) to see what they think?

[ishaan-jaff]

sure, I can make an issue there. Feel free to close this issue too

[krrishdholakia]

hey @RobertCraigie is there a workaround we can implement to bump the httpx version and support multilingual requests?

[RobertCraigie]

I haven't looked into this myself and I can't find @ishaan-jaff's issue on the httpx repo (did you open one @ishaan-jaff?).

I would recommend opening an issue with httpx / asking them for guidance. You may be able to manually encode characters yourself with event hooks as a workaround.