Skip to content

fix(darwin): framework-ize ctypes .dylib libs so they load on the iOS simulator#223

Open
ndonkoHenri wants to merge 1 commit into
mainfrom
fix/darwin-ctypes-dylib-xcframework
Open

fix(darwin): framework-ize ctypes .dylib libs so they load on the iOS simulator#223
ndonkoHenri wants to merge 1 commit into
mainfrom
fix/darwin-ctypes-dylib-xcframework

Conversation

@ndonkoHenri

Copy link
Copy Markdown
Collaborator

Problem

A Python package that ships plain .dylib shared libraries loaded via ctypes (rather than a CPython .so extension) fails to dlopen on the iOS simulator:

dlopen(.../llama_cpp/lib/libllama.dylib): mach-o file (...), but
incompatible platform (have 'iOS', need 'iOS-simulator')

C-extensions (numpy, stdlib) work fine on the simulator; only .dylib-shipping packages fail.

Root cause

The darwin site-packages sync framework-izes only *.so files (xcframework_utils.sh: dylib_ext=so). Each .so becomes a device+simulator xcframework + a .fwork pointer, and Xcode extracts the correct per-slice binary — that's why C-extensions load on both device and simulator. Plain *.dylib files are skipped entirely and copied straight from the device base (sync_site_packages.sh copies archs[0] = iphoneos.arm64), so a simulator app ends up with a device-platform dylib.

The simulator/device intent never reaches this layer: flet build ipa and flet build ios-simulator both call serious_python … --platform iOS; they only diverge later at flutter build ios --simulator. So the fix has to make the .dylib path platform-correct on its own.

Fix (2 files, serious_python_darwin/darwin/)

  • sync_site_packages.sh: iterate both so and dylib in the conversion loop, passing the extension to the helper.
  • xcframework_utils.sh: take the extension as a parameter and use it for the simulator-slice globs, so .dylib libs get the same device+fat-simulator xcframework + .fwork as .so.
  • Preserve the install-name for .dylib (gate the install_name_tool -id rewrite on ext=so). This is the key to multi-lib ctypes packages: e.g. libllama has LC_LOAD_DYLIB @rpath/libggml.dylib. Keeping the install-name lets a wrapper that preloads its dependency libs with RTLD_GLOBAL satisfy those loads via dyld's already-loaded-image match. Rewriting the id to the framework path (the .so behavior) would break that.

Non-regression / blast radius

  • The .so path is byte-for-byte unchanged (ext=so keeps the id-rewrite) — verified numpy/stdlib frameworks are identical.
  • The change only activates on .dylib. The flet-lib* ctypes recipes all ship a single unversioned lib*.so on purpose, so nothing shipping today produces a .dylib — this is purely additive.
  • Guards for future .dylib recipes: -type f skips SONAME symlink chains, and conversion is skipped when a lib has no simulator slice (avoids an empty lipo), leaving it as-is.

Testing

  • llama-cpp-python on the iOS simulator (arm64): import + native ctypes calls + real GGUF inference all pass (previously the platform-mismatch dlopen failure). Verified against a normally-built app on both the published 1.0.1 darwin scripts and this branch's scripts.
  • .so non-regression: numpy frameworks unchanged (id still rewritten to the framework path).
  • Device slice validated statically from the intermediate xcframework (ios-arm64: platform 2, install-names preserved, inter-lib deps intact). Device runtime / code-signing not verified on hardware — the .dylib→framework path uses the same Xcode embed/sign as the existing .so framework path.

Proposed CHANGELOG change

* darwin: also package ctypes `.dylib` shared libs as per-slice xcframeworks on
  iOS (previously only `.so` C-extensions), so they load on the simulator;
  preserve their install-name so multi-lib packages resolve their siblings.
… simulator

The darwin site-packages sync only converted Python C-extension *.so files
into device+simulator xcframeworks (dylib_ext=so); plain *.dylib shared libs
(the kind a ctypes wrapper dlopens — e.g. llama-cpp-python's libllama +
libggml*) were skipped and shipped straight from the archs[0]=iphoneos DEVICE
slice. On the simulator that dylib then fails to dlopen:

    mach-o file (...libllama.dylib), but incompatible platform
    (have 'iOS', need 'iOS-simulator')

C-extensions pass because their .so is framework-ized into a fat (device +
simulator) xcframework and Xcode extracts the right per-slice binary; .dylib
libs never got that treatment.

Teach create_xcframework_from_dylibs to take the file extension as a parameter
and iterate both `so` and `dylib` in sync_site_packages.sh, so .dylib libs also
become a device+fat-simulator xcframework + a .fwork pointer, exactly like .so.

For .dylib, PRESERVE the original install-name (gate the `install_name_tool -id`
rewrite on ext=so). Multi-lib ctypes packages — libllama depends on
@rpath/libggml*.dylib — then resolve their siblings: the wrapper preloads the
dependency libs with RTLD_GLOBAL and dyld matches them by their preserved
install-name. The .so path is untouched (ext=so keeps the -id rewrite), so
C-extensions (numpy, stdlib) are byte-for-byte identical.

Guards: `-type f` skips SONAME symlink chains; conversion is skipped when a lib
has no simulator slice (avoids an empty `lipo`), leaving it as-is.

Tested with llama-cpp-python on the iOS simulator: import + native ctypes calls
+ real GGUF inference all pass (previously failed with the platform mismatch).
numpy (.so) unchanged; the device xcframework slice was validated statically
(platform 2, install-names preserved, inter-lib deps intact).

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Updates the Darwin site-packages packaging pipeline so ctypes-loaded .dylib libraries (not just CPython .so extensions) are converted into per-slice xcframework-backed frameworks for iOS, allowing them to load correctly on the iOS simulator.

Changes:

  • Extend site-packages conversion to process both *.so and *.dylib artifacts during iOS staging.
  • Parameterize the xcframework helper by extension and skip install_name_tool -id rewriting for .dylib to preserve install-names.
  • Add a guard to leave device-only artifacts unconverted when no simulator slice exists.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File Description
src/serious_python_darwin/darwin/xcframework_utils.sh Adds extension-parameter support, simulator-slice existence guard, and conditional install-name rewriting for .so only.
src/serious_python_darwin/darwin/sync_site_packages.sh Expands the conversion loop to framework-ize both .so and .dylib files into xcframeworks during iOS packaging.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/serious_python_darwin/darwin/xcframework_utils.sh
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

2 participants