Build and host pre-built artifacts for python-for-android and kivy-ios

[misl6]

During the last few days, I've worked alongside @T-Dynamos (who is doing a great job in maintaining python-for-android and migrating recipes to isolated builds) to write this proposal.

Feel free to share thoughts or send suggestions.

Preface

During the past few years, users were forced to build their dependencies from scratch, and that made quite un-easy for newcomers to hike the Mount Everest of “cross compiling”.

This issue does not only applies to non-plain-python python packages, but also (and not limited to) libraries such as sdl2, libffi or sqlite3 and our beloved python3 interpreter.

What was basically forcing us (or making it harder) to avoid having pre-build artifacts is the inability to build python packages in an isolated mode.

However, thanks to a contributor (@Anshdadwal) awesome work, we’re now able (at least on python-for-android) to build python packages in an isolated environment.

Additionally, PEP 730 and PEP 738 will make things easier quite soon, so now seems the best time to start working on building and hosting pre-build artifacts.

The current status

Both python-for-android and kivy-ios are specifically made to being run on the developer machine, and even if testing is made on CI/CD side, we do not have a proper workflow to get the build and store artifacts, so mostly everything needs to be done from scratch.

The proposal

Phase 1: Separating build from installation, and creating proper install archives

ATM all the produced artifacts are automatically installed. We do not have a clean separation between the build and the installation phase, even if we tried to do so in code (sometimes).

The build and the install part should be separated clearly, so:

• build: Takes care of patching, cross-compiling, and creating the archive.
• install: Takes care of unpacking the archive, and placing headers, files, libraries to the proper place.

archive format: A .zip or .whl (for wheels, no need to re-invent the wheel 😅) file.

The .zip artifact should have the following folder structure:

metadata/
	properties.json
	install.json
data/
	file1.txt
	file2.tar.gz
	file2.whl
	anything-you-want-dir

metadata/properties.json is a json file which stores all the package properties, such as:
- stage: aka the python-for-android branch (master, develop, RFU (such as backports support for old python versions)
- name: extracted from recipe name - str
- arch: Options may differ based on the target platform (iOS or Android). In case of iOS, it may even be a multi-arch label.
- native_api_level: minimum supported api version (ndk version for Android, min supported iOS version for iOS)
- kind: defined by recipe kind lib, python, files
- version: extracted from pinned package version in recipe - str
- release_version: RFU - str
- lib_dependencies: list, should be a valid lib
- files_dependencies: list, should a valid files artifact.

metadata/install.json is a a json file which stores a list of install instructions, which are executed from top to bottom.
Install instructions are composed of install primitives (ATM only copy is available) and additional options:
- copy: Used to copy files from the artifact to the install folder.
- options:
- src: Mandatory, relative to artifact data folder
- dst: Mandatory, may contain placeholders such as $PROJECT_FOLDER, $ANDROID_LIB_FOLDER, etc. (todo: define placeholders)

data: You can store basically anything into the data folder, and you can follow your own structure.

archive name: stage-(lib,python,files)-name-arch-native_api_level-version.zip

Different artifact kinds are installed in the following order:

• libs
• python
• wheels
• files

Once the CI/CD pipeline notices a change in the recipes/name folder the package is built and saved as an artifact result on Github PR, so we should define a common path to place built artifacts.

Phase 2: Automated build. Manual upload. Manual test.

We now have proper storable and installable artifacts, but the build & install phase still happens on the same machine.

During this phase, we will allow users to download artifacts from a publicly available storage, however, the artifacts upload will be performed manually when a maintainer (or contributor with additional privileges) decides to perform a release. Only the develop stage will be available.

Additionally, to avoid incurring in high-bandwidth requirements (and abuse), users should manually download and place files in a folder where python-for-android or kivy-ios can later retrieve it. That means no automation is still included in both projects, but we will configure both to check for pre-built packages in a cache directory.

This phase will allow us to perform tests (check that the builds effectively work as expected when used on a different machine, version check, dependency tree, etc …).

Phase 3: Automated build. Automated upload. Manual Test.

Time to build a “bot”, to handle releases. Artifacts are always created, but should be uploaded to the distribution storage only when a release is requested by someone with the proper permissions.

The current CI/CD pipeline rules are just not enough as we need to specify which artifact(s) should be uploaded.

To do so, inside every recipe folder, a release.json file will be added. This file will keep track of every recipe release,

The release.json file will have the following content:

{
	"version": "1.0.0" # Should be the same as the one tracked down into the recipe
	"build": "0" # should be set to 0 if "version" changes.
	"release": true # should be set to false if "version" or "build" has been changed but no release should be performed (e.g. WIP but merged)
}

Once the CI/CD pipeline notices a change in the recipe/name/release.json after a PR merge, the CI/CD should try to upload the artifact to the distribution server (the upload process will be similar to the one currently in use by kivy/kivy)

A stage.json file will be added on the root folder of the project. That file will contain information regarding the current stage, which can be used during CI/CD operations or build.

Phase 4: All automated.

Before this phase, we will work towards our current hosting sponsors or look out for new partnerships, to make sure the Kivy infrastructure can sustain continuous downloads from the pre-built package storage.

python-for-android, and kivy-ios will be configured to look-up for pre-built packages from the selected stage.

If no pre-built package is available, but a recipe is, python-for-android (or kivy-ios) will try to self-build that package.

About stages

Each stage will have its own endpoint, which is allocated manually upon request by a Kivy Core Developer team member (or by someone with the proper privileges to access the Kivy web infrastructure).

kivy-ios and python-for-android will have two different environments, so stages and other details can be configured differently.

The stage should reflect a valid branch of the python-for-android (or kivy-ios) project, so can be maintained / freezed based on the development needs.

Wheels and non-wheels will have two separate environments for the same stage.

Every stage will have a total of 3 public endpoints:

• https://stage-name.authority.android-packages.kivy.org :
  • Will be the authoritative entity which provides info regarding a specific package (version, build, …)
• https://stage-name.wheels.android-packages.kivy.org
  • Simple repository for python packages (https://packaging.python.org/en/latest/guides/hosting-your-own-index/), so wheels installs can be later managed via python-provided tools.
• https://stage-name.storage.android-packages.kivy.org
  • Storage box for libraries and files.

About version and build management

Users will have the opportunity to pin to a specific version, if available, of a specific dependency (lib, files or wheel).

As seen previously, version should reflect the version of the library / tool / wheel which is being built by the recipe.

However, mistakes can happen, and we may ask users to download and re-install a specific library (or wheel). To do that, the maintainer, should bump the build number of the relevant release.json file, so a new release will be performed.

On python-for-android (or kivy-ios side), the look-up, download and install logic should check if the build version have changed, and act accordingly. The upgrade can be automated, or an interaction may be required from the user, based on the user provided options.

About supported archs and api levels

Obviously, we can’t store and officially support all the combinations of archs and api levels.
Most of the users, even right now, are just using the default NDK target api level, or on iOS the default supported version.

So, even if the new structure and changes will allow users to self-build artifacts with a NDK api level different from the one which will be made available for direct download, we will not store or produce artifacts for these “officially unsupported” versions.

On Android, only artifacts for armeabi-v7a, arm64-v8a and x86_64 will be produced and stored, as x86 artifacts are considered obsolete and a niche.

[Sahil-pixel]

So then Build for Android and iOS will be very fast. Because packages are already complied. Great Idea. 😀

[Sahil-pixel]

I am so excited . To use that . 😀😀😀😀

[AccelQuasarDragon]

Nothing real to add, great change, seconded!

[cederom]

• very cool to bring automation and packages in order to speed up the mobile app generation!
• this will be an option and users will still be able to build all from scratch right?
• some sort of checksums and digital signatures would be welcome for a released packages so we could confirm their coherence and origin :-)

[T-Dynamos]

this will be an option and users will still be able to build all from scratch right?

Yes

some sort of checksums and digital signatures would be welcome for a released packages so we could confirm their coherence and origin :-)

Thanks for suggesting, this should be automatically done while packages are being downloaded.

[Sahil-pixel]

We need all possible way . To build fast build which is already compiled. And build by compiling source. Which gives full flexibility in build process

[tito]

Did you study the wheel file format? Could it not possible to use it directly instead of making a new intermediate format?

[mhsmith]

We're currently using the wheel format for non-Python libraries in Chaquopy for Android – see the "chaquopy-" directories here. Each of those wheels contains a lib and include directory so it can be used both at runtime and for building other packages against it. Dependencies and other metadata are stored in the standard wheel format (.dist-info/METADATA).

However, this approach can't easily be migrated to PyPI, because being a Python-only repository, it's not clear who would take responsibility for packaging all of those non-Python libraries. Instead, I'd like to eliminate the non-Python "chaquopy-" packages, and bundle those libraries within the Python wheels that use them, as is currently done with manylinux wheels with the help of the auditwheel tool.

Where the builders of the Python packages would get the non-Python libraries from is TBD. I've thought of using a non-Python-specific platform such as conda-forge, which has pretty good support for cross-compilation, semi-automatic version updates, automatic building, and reliable hosting.

[freakboy3742]

It's worth noting that this problem is essentially why Conda exists - the need for a consistent distribution story for an entire binary ecosystem, not just the Python code in that ecosystem. It is possible to install wheels into a Conda environment; but the more reliable approach, especially for binary wheels, is to develop a Conda recipe so that all binary library dependencies are consistently linked.

Rather than try to work out how to distribute a non-Python binary package as a Python wheel (which, as @mhsmith notes, is possible, but not really endorsed by PyPI), it may be better to look at that class of problem as a "Conda problem", and work within that ecosystem to add support for Android and iOS as target platforms.

(Full disclosure: @mhsmith and I both work for Anaconda; Conda and Conda-Forge are open source projects that Anaconda supports)

[T-Dynamos]

bundle those libraries within the Python wheels

But wouldn't this increase size of overall app if multiple packages use same library like libc++? Then we might need some sort of post cleaning of site-packages directory to check for same libraries and make symlinks after removing duplicate.

[freakboy3742]

But wouldn't this increase size of overall app if multiple packages use same library like libc++?

Yes - but that's how Python wheels work. They duplicate dependent libraries when the system can't be relied on to provide them. You could definitely attempt some sort of post-install cleanup - but that's not done by pip (or any other tool I'm aware of in the wheel ecosystem).

FWIW, it's not an issue that exists with conda - specifically because non-Python binary dependencies are declared as a dependency, rather than being bundled.

[mhsmith]

The last time I investigated auditwheel was when I wrote this StackOverflow answer. In summary, it bundles the libraries into a package_name/.libs directory:

 38513408  2017-10-25 06:02   scipy/.libs/libopenblasp-r0-39a31c03.2.18.so
  1023960  2017-10-25 06:02   scipy/.libs/libgfortran-ed201abd.so.3.0.0

Another option would be to put the bundled libraries into a top-level .libs directory, shared by all wheels. Since the filenames contain a hash, this means that when multiple wheels provide identical libraries, they would be deduplicated automatically at install time, with no need for a post-install cleanup. Different versions of the same library could still coexist.

There's also a special case for libraries like OpenSSL and SQLite that are used by Python's own standard library. We're publishing binaries of these through a GitHub repository, and including them with the Python installation. So they never need to be packaged into wheels.

[mhsmith]

I just wanted to let you know about the related work we're doing in the BeeWare project. Myself and @freakboy3742 spent a large part of 2024 implementing PEP 730 and PEP 738 to add official Android and iOS support to Python 3.13, and now we're turning our attention to the third-party package situation.

BeeWare and Chaquopy have been building and hosting mobile platform wheels for a few years. They can be found here:

• Android: build tool, repository
• iOS: build tool, repository

Many of our patches were originally based on Kivy's, so thank you for that.

However, we've found that user demand for new and updated packages is too much for us to keep up with. So we're working on adding support for mobile platforms to general-purpose packaging tools such as cibuildwheel. The plan is that our existing tools and repositories will eventually become obsolete, and projects will be able to build Android and iOS wheels with their existing workflows, and publish them on PyPI in the same way as Linux, Mac and Windows. I believe this is the only truly scalable solution.

Since the Python on mobile community is still quite small, I hope BeeWare and Kivy can combine our resources in this area to help build a shared package ecosystem.

[misl6]

Hi @mhsmith!



Combining resources in Open Source Software is crucial for efficient and future-proof projects, especially in a “niche” (I hate this word) like ours. So, I agree we should start doing that from the beginning. (By the way, thanks a lot for PEP 730 and PEP 738! Those who have attended one of Kivy’s talks at conferences in late 2023 and 2024 know how much I’ve fanboyed these two PEPs 😂).

Unfortunately, we haven’t had the chance to take advantage of them yet (I did some testing on iOS, though). Some major changes are needed in our recipe (our set of instructions for cross-compiling to Android and iOS) build system and load scripts, especially for iOS, where we historically statically linked everything to the main executable.

Right now, Kivy’s main focus is on the 3.0.0 release, which requires a lot of work. So, for now, both python-for-android and kivy-ios aren’t getting the attention they deserve.

Said that you can now imagine how much I agree with your “is too much for us to keep up with”. There are a lot of feature requests on both kivy-ios, python-for-android issue trackers, and related chats asking for support for a specific package. Some of them might be easy to add, but others might not be.

I think it’s better to start from scratch and build a scalable solution that works for all the “Python on mobile” ecosystem. We can take the best from our existing implementations and ideas.

I don’t think working (only/mainly) on a “cibuildwheel” implementation for mobile platforms will be enough to persuade package maintainers (who usually have limited time) to get their packages ready for mobile platforms. They’ll inevitably run into issues they’ve never seen before.



IMHO is best to start with a set of carefully selected tools, documentation, workshops and learning resources. We can create a collectively-managed (Python.org, BeeWare, Chaquopy, Kivy.org, and everyone else have something to add) recipe-like environment, at least until package maintainers are more familiar with mobile platforms and we’ve figured out how to make the process as smooth as possible.

It’s also important to keep an eye on “building from scratch.” Downloading pre-built dependencies to include in an app might not work for certain regulated industries. Package maintainers will need a way to quickly build and test their changes without rebuilding the whole package and a tool which is only meant to run in a CI/CD pipeline isn’t the best way to do that.

As someone stated, on mobile apps we should also work to keep the size as smallest as possible, and bundling the same (or almost the same [minor bugfixes]) shared library multiple times is definitely not good. A custom “manylinux-like” environment, which provides a configurable set of pre-built dependencies to use for both cross-compiling and to include into the App might circumvent this issue.



Let’s discuss, connect (online and IRL), brainstorm, and make Python the go-to language for app development!

[freakboy3742]

Downloading pre-built dependencies to include in an app might not work for certain regulated industries. Package maintainers will need a way to quickly build and test their changes without rebuilding the whole package and a tool which is only meant to run in a CI/CD pipeline isn’t the best way to do that.

While cibuildwheel might suggest that it can only run in CI, it can also be used for local wheel building. It provides the tooling for cross-platform build configuration, including multiple Python versions, platforms and architectures; and includes tooling to start the app's test suite.

I've already got a patch for cibuildwheel that adds iOS support; essentially you can run:

pip install cibuildwheel
CIBW_PLATFORM=iphonesimulator cibuildwheel

in the root of a project source tree, and it will build a binary wheel for the iOS simulator (at present, only Python3.13... but that's because that's the only version of Python that formally supports iOS), and run the test suite for the library. I've confirmed it works on a handful of packages (including Pillow). I'm maybe a week or two away from being able to submit this for formal acceptance in cibuildwheel.

A custom “manylinux-like” environment, which provides a configurable set of pre-built dependencies to use for both cross-compiling and to include into the App might circumvent this issue.



I disagree. That "manylinux-like" environment already exists - it's called "Android" and "iOS".

It's important to remember that the purpose of Manylinux is to accommodate that "Linux" isn't an operating system - it's only a kernel. But nobody uses just the kernel - they have a C library, and a bunch of other libraries. Manylinux is a standardisation attempt to say "you can assume this basic set of libraries exist" when users are actually using Fedora, Debian, Ubuntu, or whatever else.

But in the case of both Android and iOS, we know what set of libraries exist. It's the set of libraries that comes in an Android/iOS release, and ships with the standard Android NDK/iOS SDK. This is the same situation that exists on macOS and Windows - if I say a wheel is macOS 11 compatible, I know what libraries are available on macOS 11.

You might argue that we could include OpenSSL, libSDL, or some other significant libraries in a manyandroid specification - to which I would point out that manylinux doesn't do this. It's a really bare bones - really not much more than enough to guarantee a binary will start. For example, it doesn't include anything related to desktop environments - which is why pygobject and other GTK Python libraries can only be distributed as a source.

Let’s discuss, connect (online and IRL), brainstorm, and make Python the go-to language for app development!

🎉

[misl6]

You might argue that we could include OpenSSL, libSDL, or some other significant libraries in a manyandroid specification - to which I would point out that manylinux doesn't do this. It's a really bare bones - really not much more than enough to guarantee a binary will start. For example, it doesn't include anything related to desktop environments - which is why pygobject and other GTK Python libraries can only be distributed as a source.

Yeah, you got the point, I was absolutely talking about "extras", and the "manylinux" might not have fitted quite well here 😄.
Obviously we do not need to replicate what Android and iOS are already providing by themselves.

[misl6]

Addition: These "extras" are not mandatory, a package mantainer might decide to force bundling a specific (patched, self-built, or from other source) library, but by using a "common extras repository" this will be highly discouraged.