Hatchling preparing metadata fails with UTF-16 readme; assumes UTF-8

[nyoma-diamond]

When trying to build / install a project (i.e., using pip install [-e] .), preparing editable metadata fails when a readme is included in pyproject.toml where the readme is in UTF-16 (as opposed to UTF-8). The corresponding error produced is unclear and not easily parsable to recognize the issue is caused by the file being in UTF-16.

Version Details

OS: Windows 11 Pro Version 10.0.26100 Build 26100
Python: 3.12.7
Hatchling not installed manually (used via pyproject build-system). Believed version 1.27.0

Steps to Reproduce

1. Create a project with a readme file (e.g., README.md) encoded in UTF-16
2. Create a pyproject.toml that includes the readme. E.g.,

   [build-system]
   requires = ["hatchling"]
   build-backend = "hatchling.build"
   
   [project]
   name = "foo"
   readme = "README.md"

3. Attempt to install the project via pip; i.e., pip install [-e] .

Produced Error

(User/project details anonymized)

$ pip install -e .

Obtaining file:///C:/Users/username/Documents/project
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build editable ... done
  Installing backend dependencies ... done
  Preparing editable metadata (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Preparing editable metadata (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [31 lines of output]
      Traceback (most recent call last):
        File "C:\Users\username\Documents\project\.venv\Lib\site-packages\pip\_vendor\pyproject_hooks\_in_process\_in_process.py", line 195, in prepare_metadata_for_build_editable
          hook = backend.prepare_metadata_for_build_editable
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      AttributeError: module 'hatchling.build' has no attribute 'prepare_metadata_for_build_editable'

      During handling of the above exception, another exception occurred:

      Traceback (most recent call last):
        File "C:\Users\username\Documents\project\.venv\Lib\site-packages\pip\_vendor\pyproject_hooks\_in_process\_in_process.py", line 389, in <module>
          main()
        File "C:\Users\username\Documents\project\.venv\Lib\site-packages\pip\_vendor\pyproject_hooks\_in_process\_in_process.py", line 373, in main
          json_out["return_val"] = hook(**hook_input["kwargs"])
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "C:\Users\username\Documents\project\.venv\Lib\site-packages\pip\_vendor\pyproject_hooks\_in_process\_in_process.py", line 204, in prepare_metadata_for_build_editable
          whl_basename = build_hook(metadata_directory, config_settings)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "C:\Users\username\AppData\Local\Temp\pip-build-env-vqj6s20u\overlay\Lib\site-packages\hatchling\build.py", line 83, in build_editable
          return os.path.basename(next(builder.build(directory=wheel_directory, versions=['editable'])))
                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "C:\Users\username\AppData\Local\Temp\pip-build-env-vqj6s20u\overlay\Lib\site-packages\hatchling\builders\plugin\interface.py", line 90, in build
          self.metadata.validate_fields()
        File "C:\Users\username\AppData\Local\Temp\pip-build-env-vqj6s20u\overlay\Lib\site-packages\hatchling\metadata\core.py", line 266, in validate_fields
          self.core.validate_fields()
        File "C:\Users\username\AppData\Local\Temp\pip-build-env-vqj6s20u\overlay\Lib\site-packages\hatchling\metadata\core.py", line 1366, in validate_fields
          getattr(self, attribute)
        File "C:\Users\username\AppData\Local\Temp\pip-build-env-vqj6s20u\overlay\Lib\site-packages\hatchling\metadata\core.py", line 534, in readme
          self._readme = f.read()
                         ^^^^^^^^
        File "<frozen codecs>", line 322, in decode
      UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

Comments

Converting the readme to UTF-8, if possible, solves the issue. However, the provided error does not suggest to do this or indicate that anything is wrong with the file encoding (e.g., that it is UTF-16 and/or that hatchling does not support it).

Ideally hatchling should either alert that only UTF-8 is supported for readmes or be capable of supporting encodings beyond UTF-8.

I also tested this with license files, and did not run into the same issue; license files can have other encodings (that said, I don't believe hatchling processes license files to the same extent as readmes, so this may not be relevant).