Studio: GPU memory configuration for GGUF models - #6414
Merged
danielhanchen merged 157 commits intoJul 19, 2026
Conversation
# Conflicts: # studio/backend/core/inference/llama_cpp.py
…L, loadedGpuMemoryFields helper)
…l leak, validate, types)
… floor, comment accuracy)
…ooltip, shorten fit dropdown label
…or-nothing toggle
# Conflicts: # studio/frontend/src/features/chat/chat-settings-sheet.tsx # studio/frontend/src/features/chat/stores/chat-runtime-store.ts
danielhanchen
added a commit
to shimmyshimmer/unsloth-staging-4
that referenced
this pull request
Jul 19, 2026
Member
|
@codex review |
![chatgpt-codex-connector[bot]](https://cdn.statically.io/img/avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 66e6b3e8aa
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Comment on lines
+8505
to
+8508
| # A changed GPU pick must reload (compare order-insensitively; None/[] | ||
| # both mean automatic). | ||
| if (self._gpu_ids or None) != (sorted(gpu_ids) if gpu_ids else None): | ||
| return False |
There was a problem hiding this comment.
When a Diffusion GGUF is loaded with a multi-GPU gpu_ids request (which the picker permits), _start_diffusion_server intentionally records only the lowest selected GPU because the runner is single-device, but this comparison still expects the entire original list. Consequently a repeated /load or settings Apply with the same multi-GPU request cannot deduplicate and unloads/restarts the diffusion server each time. Normalize the requested pick to its effective single diffusion GPU before comparing (and in the route-level matcher).
Useful? React with 👍 / 👎.
Member
There was a problem hiding this comment.
Fixed in 958ccb1. The diffusion runner records only its single lowest device (self._gpu_ids = [sorted(gpu_ids)[0]]), so both _already_in_target_state and the route _request_matches_loaded_settings now collapse the requested pick to [sorted(gpu_ids)[0]] for a loaded diffusion model before comparing. A repeated multi-GPU request that resolves to the same device now deduplicates instead of restarting the server. Covered by test_gpu_ids_reload_detection_collapses_diffusion_to_single_device and test_route_matches_loaded_settings_collapses_diffusion_gpu_ids.
Comment on lines
+270
to
+279
| if single_device_gpu is not None: | ||
| try: | ||
| selected_gpu = int(single_device_gpu) | ||
| if selected_gpu < 0: | ||
| raise ValueError | ||
| except (TypeError, ValueError): | ||
| return False, { | ||
| "mode": "single_device", | ||
| "reason": "unresolved_gpu_id", | ||
| } |
There was a problem hiding this comment.
When training is active and DG_GPU is configured with a supported nonnumeric token such as a CUDA UUID (or is empty to force CPU), _diffusion_gpu_arg forwards that token here, but can_load_chat_during_training converts it with int() and returns unresolved_gpu_id. The guard therefore rejects every such diffusion load with 409 before the runner starts, even though the runner itself receives and uses the same DG_GPU token. Resolve UUID/CPU tokens against visible devices or fall back to the existing GGUF guard instead of treating them as an invalid integer.
Useful? React with 👍 / 👎.
Member
There was a problem hiding this comment.
Fixed in 958ccb1. can_load_chat_during_training no longer calls int() blindly on the single-device token: an empty token (a CPU-only runner such as a CPU diffusion GGUF) is allowed outright since it uses no GPU VRAM, and a non-numeric token such as a CUDA UUID or MIG handle falls back to sizing against the whole visible pool like the GGUF guard instead of returning unresolved_gpu_id. Covered by test_single_device_unresolved_token_sizes_against_pool, test_single_device_unresolved_token_refuses_when_pool_full, and test_single_device_cpu_token_allows.
The diffusion runner drives only its single lowest device and the backend records that one device (self._gpu_ids = [sorted(gpu_ids)[0]]), but the reload dedupe compared it against the full requested list, so a multi-GPU pick that resolves to the same device forced a needless reload. Normalize the request the same way for a loaded diffusion model in both _already_in_target_state and the route _request_matches_loaded_settings. The chat-during-training coexistence guard called int() on the single-device token and hard-rejected when it could not parse. A non-numeric token (a CUDA UUID / MIG handle) now sizes against the whole visible pool like the GGUF guard instead of falsely blocking the load, and an empty token (a CPU-only runner such as a CPU diffusion GGUF) is allowed outright since it uses no GPU VRAM.
Member
|
@codex review |
|
Codex Review: Didn't find any major issues. Already looking forward to the next diff. Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
danielhanchen
added a commit
to shimmyshimmer/unsloth-staging-4
that referenced
this pull request
Jul 19, 2026
… TP reset, tensor_split validation - Training coexistence guard: a single-device runner pinned through an unresolvable UUID/MIG token was sized against the aggregate visible-VRAM pool, so a load could pass on capacity it cannot use and then OOM active training. Size against the worst-case visible device (min free) instead, keeping the guard's documented default-deny contract. The empty-token (CPU-only runner) allow path is unchanged. - Diffusion startup: _start_diffusion_server now resets self._tensor_parallel to False alongside the other placement resets. A prior tensor-parallel chat load (process killed but not fully unload-reset) otherwise left /status misreporting tensor parallelism and made an identical diffusion re-Apply reload against the stale state. - tensor_split: reject negative / non-finite / all-zero splits up front. They were dropped at launch but still compared raw in the reload dedupe, so an identical Apply reloaded indefinitely. - Tests: the shared httpx stub was incomplete and, installed via setdefault before real httpx loaded, broke a combined pytest run (collection errors on httpx.Response). Import the real installed httpx instead.
Member
|
I ran an independent multi-reviewer pass over this PR and triaged the results against the code. Pushing fixes for the safe, high-value items; noting the rest so you can decide. Fixed in this push:
Noted, not changed here (lower severity or your design call):
Checked and dismissed as false positives:
All backend suites pass per-file, plus the combined run is now clean. Happy to send any of the "noted" items as a follow-up if you'd like. |
for more information, see https://pre-commit.ci
Member
|
@codex review |
|
Codex Review: Didn't find any major issues. 🚀 Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
This was referenced Jul 19, 2026
danielhanchen
added a commit
to shimmyshimmer/unsloth-staging-4
that referenced
this pull request
Jul 19, 2026
danielhanchen
added a commit
that referenced
this pull request
Jul 20, 2026
…location (#7252) #6414 moved the llama_extra_args inheritance out of the GGUF branch in _load_model_impl into _guard_chat_load_against_training, which runs before the branch, so 'if request.llama_extra_args is None' is no longer inside the gguf_branch slice that test_load_marker_precedes_hub_guard_and_unload checks. The assertion failed on that now-missing landmark even though the guarantee it protects (the gguf_load_in_flight marker is entered before the hub-download guard and the unload) is intact. Drop the relocated landmark from the ordering so the test matches the current structure. Co-authored-by: danielhanchen <unslothshared@gmail.com>
VectorCipher
pushed a commit
to VectorCipher/unsloth
that referenced
this pull request
Jul 20, 2026
* Studio: GPU memory dropdown — llama.cpp --fit on and manual gpu-layers/cpu-moe * Studio: simplify GPU memory changes (reuse ParamSlider, GPU_LAYERS_ALL, loadedGpuMemoryFields helper) * Studio: GPU picker — choose which GPUs a GGUF model loads on (gpu_ids) * Studio: simplify GPU picker (share /api/system fetch, validate gpu_ids) * Studio: GPU picker review fixes (gate relative indices, no cross-model leak, validate, types) * Studio: group GPU controls under a collapsible GPU section * Studio: GPU feature review fixes (fix fit-ctx test, behavior-test the floor, comment accuracy) * Studio: make GPU a top-level settings section (not nested under Model) * Studio: flatten GPU controls into the Model section, group by GPU/context/generation * Studio: move GPU Memory to the bottom of Model with its dependent controls beneath it * Studio: move GPU Memory below Tensor Parallelism and GPUs below GPU Memory * Studio: tighten GPU Memory and GPU Layers tooltip copy * Studio: fix fit-mode context slider track-click, restore GPU Memory tooltip, shorten fit dropdown label * Studio: GPU Memory tooltip one mode per line, briefer * Studio: note HIP_VISIBLE_DEVICES (ROCm) in the GPUs picker tooltip * Studio: narrow the GPU Memory dropdown to fit the shortened label * Studio: use 'llama.cpp --fit' in the GPU Memory tooltip for consistency * Studio: allow Tensor Parallelism in Manual GPU mode * Studio: graduated MoE-on-CPU offload (--n-cpu-moe) replacing the all-or-nothing toggle * Studio: size the MoE-offload slider for staged (deferred-load) models * Studio: share one GGUF header walk for the context-length and MoE-count readers * Studio: size the GPU Layers slider for staged models (one staged-header read) * Studio: move Tensor Parallelism below the GPUs picker * Studio: GPU split (--tensor-split) per-GPU model share in Manual mode * Studio: tolerate whitespace in GPU split input, move it below GPU Layers * Studio: rename the GPU split control to "Split ratio" * Studio: Split ratio sends explicit even input; fix blank=free-VRAM (not even) copy * Studio: tighten llama.cpp --fit VRAM margin with --fit-target 512 * Studio: GPU memory review fixes (rollback re-baseline, single-GPU TP gate, accurate copy) * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: move Split ratio below MoE Layers on CPU * Studio: address PR review (fix GPU-info hydration race, share fit context-length across load paths) * Studio: address codex review (manual single-GPU TP guard, GPU-aware spec defaults in fit/manual, GGUF-only context/preference) * Studio: address codex review round 2 (gpu_present seed, single-GPU tensor-split guard, staged manual-knob reset, strip inherited offload flags) * Studio: address codex review round 3 (strip inherited --n-cpu-moe, CPU-fallback warning in Manual mode) * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: address codex review round 4 (preserve pinned fit context across a later Apply) * Studio: address codex review round 5 (honor GPU picker for diffusion GGUFs, clear fit pin on cross-model switch) * Studio: preserve the pending GPU Memory mode when staging a model * Studio: pin diffusion GPU device order and reset GPU-memory state for diffusion loads * Studio: address codex review round 6 (fit-Auto rollback context, preserve manual non-tensor split modes, persist GPU mode on load not select) * Studio: persist the applied GPU Memory mode, not the requested one (skip diffusion loads) * Studio: replace Manual-mode split-ratio field with per-GPU layer sliders * Studio: clarify per-GPU layer split hint for tensor-parallel mode * Studio: address codex review round 7 (allow GGUF gpu_ids past the legacy guard, replay GPU-memory fields on respawn) * Studio: address codex review round 8 (size the validate preflight like the load in fit mode, across both load paths) * Studio: skip the training-OOM guard for llama.cpp --fit GGUF loads (they spill to RAM) * Studio: drop the now-redundant compare-path validate sizing (the --fit guard skip makes it moot) * Studio: address codex review round 9 (keep the training guard for fit loads, forward gpu_ids to validate, strip inherited manual tensor-split) * Studio: address codex review round 10 (gate GPU-memory adoption on is_gguf, record manual knobs only in Manual mode) * Studio: handle diffusion GGUFs symmetrically in the GPU Memory controls (preserve the standing mode preference, hide the inapplicable mode/TP controls) * Studio: remember the GPU Memory settings per model * Studio: consolidate --fit mode and Manual mode into a single Manual mode * Studio: preserve the per-GPU layer split across GPU Layers changes * Studio: trim overly long GPU Memory comments * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * address GPU memory config review comments * trim redundant GPU memory tests * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Reconcile manual-mode TP drops with the unslothai#6659 drop-site invariants * Preserve quantized KV in manual --fit, charge GGUF companions in full, reconcile GPU pick on load * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Clear stale GPU baseline on non-GGUF loads so it can't read as dirty * Fix no-context-shift test for the conditional -c flag * Credit manual GPU-layer offload for cached HF GGUFs * Reset per-model load knobs on GGUF quant switch * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Strip inherited tensor-split when manual ratio is cleared * Match auto-load validation to safetensors placement * Reset editable manual knobs after Auto GGUF loads * Record a single device for diffusion GPU picks * Reset per-model GPU knobs before applying saved settings * Address review comments * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Guard manual tensor splits and keep remembered context on auto-load * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Snapshot compare knobs, seed splits from free VRAM, flag zero-offload loads * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Exempt CPU-only loads from the guard floor and harden compare and reseed paths * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Reach full offload from the layers slider and charge extras drafters in the guard * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Warm the GPU device cache before pick reconciles and disable staged GPU controls * Align the training guard with inherited extras, spec mode, and compare targets * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Hide GPUs from companion-less zero-offload loads * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Size diffusion picks per device, own manual offload flags, reject XPU picks * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Drop tensor flags at zero layers and exempt CPU-pinned drafters * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Allowlist the zero-layer tensor parallel drop site * Keep validate and load guards on the same extras and refresh stale baselines * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Drop mismatched manual tensor splits before launch * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Gate XPU picks on the real backend field and harden split and hydration paths * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Weight full GPUs as zero, clamp split shares, and refine the zero-layer mask gate * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Carry fit context across mode changes and align drafter and picker gates * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Catch variant switches, uncached diffusion repos, and text-only mmproj skips * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Check companions on the first device and size native and remote zero-layer loads * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Replace the training guard's precise VRAM modeling with a conservative bound * Baseline context pins on non-GGUF hydration and reprobe list-seeded staged GGUFs * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Size manual splits by their largest share and preserve resolved context from Default * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Default-deny unsized required companions and price KV at the effective cache dtype * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Reserve MTP draft KV and MLA target-copy in the training guard * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Size tensor-parallel loads per device and show GPU controls for native GGUFs * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Reserve MTP overhead for uncached remote GGUFs and the mmproj runtime factor * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Drop the training-coexistence VRAM estimation this PR added * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Gate remembered load settings to GGUF picks * Lock the remaining load-time controls during a staged load * Clear the stale native-path token on compare loads * Drop a stale guard reference from the zero-offload masking comment * Seed GPU baselines from the rollback response and drop never-emitted offload flags * Match validate's training guard to load and keep the native reload token * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Trim verbose GPU-memory comments * Thread the variants header walk off the event loop, honor device pins on zero-offload, and hold staged GPU edits * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Honor manual placement and classify pinned zero-offload loads * Close diffusion admission and status hydration gaps * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Check the actual diffusion GPU during training * Align staged baselines and manual reload dedupe * Fix GGUF placement and rollback state * Harden manual GGUF placement boundaries * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Remove unused resolve_tensor_parallel import in llama_cpp.py The name is used only in llama_server_args.py, routes/inference.py, and tests, not in llama_cpp.py; the unused hoisted import trips the import-hoist verifier in the source-lint CI job. * Fix diffusion GPU dedup and training guard for non-numeric device tokens The diffusion runner drives only its single lowest device and the backend records that one device (self._gpu_ids = [sorted(gpu_ids)[0]]), but the reload dedupe compared it against the full requested list, so a multi-GPU pick that resolves to the same device forced a needless reload. Normalize the request the same way for a loaded diffusion model in both _already_in_target_state and the route _request_matches_loaded_settings. The chat-during-training coexistence guard called int() on the single-device token and hard-rejected when it could not parse. A non-numeric token (a CUDA UUID / MIG handle) now sizes against the whole visible pool like the GGUF guard instead of falsely blocking the load, and an empty token (a CPU-only runner such as a CPU diffusion GGUF) is allowed outright since it uses no GPU VRAM. * Tighten comments added by the GPU memory config changes * Harden GGUF placement from independent review: VRAM sizing, diffusion TP reset, tensor_split validation - Training coexistence guard: a single-device runner pinned through an unresolvable UUID/MIG token was sized against the aggregate visible-VRAM pool, so a load could pass on capacity it cannot use and then OOM active training. Size against the worst-case visible device (min free) instead, keeping the guard's documented default-deny contract. The empty-token (CPU-only runner) allow path is unchanged. - Diffusion startup: _start_diffusion_server now resets self._tensor_parallel to False alongside the other placement resets. A prior tensor-parallel chat load (process killed but not fully unload-reset) otherwise left /status misreporting tensor parallelism and made an identical diffusion re-Apply reload against the stale state. - tensor_split: reject negative / non-finite / all-zero splits up front. They were dropped at launch but still compared raw in the reload dedupe, so an identical Apply reloaded indefinitely. - Tests: the shared httpx stub was incomplete and, installed via setdefault before real httpx loaded, broke a combined pytest run (collection errors on httpx.Response). Import the real installed httpx instead. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: danielhanchen <unslothshared@gmail.com> Co-authored-by: danielhanchen <danielhanchen@gmail.com>
VectorCipher
pushed a commit
to VectorCipher/unsloth
that referenced
this pull request
Jul 20, 2026
…location (unslothai#7252) unslothai#6414 moved the llama_extra_args inheritance out of the GGUF branch in _load_model_impl into _guard_chat_load_against_training, which runs before the branch, so 'if request.llama_extra_args is None' is no longer inside the gguf_branch slice that test_load_marker_precedes_hub_guard_and_unload checks. The assertion failed on that now-missing landmark even though the guarantee it protects (the gguf_load_in_flight marker is entered before the hub-download guard and the unload) is intact. Drop the relocated landmark from the ordering so the test matches the current structure. Co-authored-by: danielhanchen <unslothshared@gmail.com>
This was referenced Jul 22, 2026
oobabooga
pushed a commit
to InfoSage05/unsloth
that referenced
this pull request
Jul 25, 2026
Narrowed to complement unslothai#6414 (merged), which already provides the GGUF GPU device picker, gpu_memory_mode (auto/manual), gpu_layers, n_cpu_moe, and tensor_split. This adds only what unslothai#6414 lacks: 1. Host-residency `gguf_memory_mode` (auto/pinned/resident) mapping to llama.cpp --mlock / --no-mmap. Distinct from unslothai#6414's gpu_memory_mode (that controls VRAM/layer offload; this controls host-RAM residency). Default auto is a no-op: an omitted value launches identical argv/env to main. First-class field shadows and scrubs inherited --mmap/--no-mmap/--mlock and LLAMA_ARG_MLOCK/NO_MMAP/MMAP so a stale value can't leak. Backend/API-driven (mirrored from /status), no new UI control. 2. Vulkan-ordinal gpu_ids hardening. unslothai#6414 rejects gpu_ids on a Vulkan build with HTTP 400; this replaces that with real pinning: gpu_ids are treated as ggml Vulkan ordinals (matched against the probe directly, never remapped through CUDA_VISIBLE_DEVICES), pinned via --device Vulkan<i>, with conflicting user --device / inherited LLAMA_ARG_DEVICE stripped under a pin, and the training coexistence guard budgeting conservatively against the least-free card (ordinals can't map to physical free-VRAM). Backend: gguf_memory_mode field + validator (LoadRequest / ValidateModelRequest, echoed on responses); _memory_mode_flags / _canonical_memory_mode; strip_memory_mode + strip_device in llama_server_args; is_vulkan_build / assert_requested_gpu_ids_resolvable; memory-mode reload-dedup in _already_in_target_state and the route matcher (device-stripped under a pin, mirroring the launch); gpu_ids_are_vulkan_ordinals budgeting in training_vram. Frontend: a read-only activeMemoryMode mirror wired into the existing reload flow (no dropdown). unslothai#6414's gpu_ids picker is untouched.
danielhanchen
added a commit
that referenced
this pull request
Jul 29, 2026
…#7188) * feat(studio): GGUF host-residency memory mode + Vulkan gpu_ids pinning Narrowed to complement #6414 (merged), which already provides the GGUF GPU device picker, gpu_memory_mode (auto/manual), gpu_layers, n_cpu_moe, and tensor_split. This adds only what #6414 lacks: 1. Host-residency `gguf_memory_mode` (auto/pinned/resident) mapping to llama.cpp --mlock / --no-mmap. Distinct from #6414's gpu_memory_mode (that controls VRAM/layer offload; this controls host-RAM residency). Default auto is a no-op: an omitted value launches identical argv/env to main. First-class field shadows and scrubs inherited --mmap/--no-mmap/--mlock and LLAMA_ARG_MLOCK/NO_MMAP/MMAP so a stale value can't leak. Backend/API-driven (mirrored from /status), no new UI control. 2. Vulkan-ordinal gpu_ids hardening. #6414 rejects gpu_ids on a Vulkan build with HTTP 400; this replaces that with real pinning: gpu_ids are treated as ggml Vulkan ordinals (matched against the probe directly, never remapped through CUDA_VISIBLE_DEVICES), pinned via --device Vulkan<i>, with conflicting user --device / inherited LLAMA_ARG_DEVICE stripped under a pin, and the training coexistence guard budgeting conservatively against the least-free card (ordinals can't map to physical free-VRAM). Backend: gguf_memory_mode field + validator (LoadRequest / ValidateModelRequest, echoed on responses); _memory_mode_flags / _canonical_memory_mode; strip_memory_mode + strip_device in llama_server_args; is_vulkan_build / assert_requested_gpu_ids_resolvable; memory-mode reload-dedup in _already_in_target_state and the route matcher (device-stripped under a pin, mirroring the launch); gpu_ids_are_vulkan_ordinals budgeting in training_vram. Frontend: a read-only activeMemoryMode mirror wired into the existing reload flow (no dropdown). #6414's gpu_ids picker is untouched. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Preserve inherited memory flags and reject GPU pins on CPU-only llama.cpp builds - strip_shadowing_flags: default strip_memory_mode to False (opt-in, matching strip_offload / strip_tensor_split / strip_device). The inherit resolver now strips an inherited --mlock/--mmap/--no-mmap only when the request supplies gguf_memory_mode, so a same-model Apply that omits the field keeps a user's pass-through memory flag instead of silently dropping it. - /load and /validate: on the CUDA path, reject explicit gpu_ids when the llama.cpp build ships no cuda/hip/vulkan ggml lib (CPU-only). Such a build ignores CUDA_VISIBLE_DEVICES, so the pin would run on CPU while the API reports it active. Mirrors the non-CUDA resolvable check. - Document that gpu_ids are ggml Vulkan ordinals on a Vulkan build (enumerated independently of CUDA_VISIBLE_DEVICES), not CUDA physical indices. - Add regression tests: CPU-only route reject, inherit preserve/strip of memory flags, and the flipped strip_memory_mode default. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Detect versioned Vulkan libs and sync memory mode across all load paths - _is_vulkan_backend now matches versioned Vulkan sonames (libggml-vulkan.so.0) via a shared _lib_dir_has_ggml_backend matcher, unified with _backend_lacks_gpu_lib so Vulkan detection and the CPU-only-build check agree. A distro/split-lib Vulkan install without the dev-only unversioned symlink is now detected as Vulkan and gets the --device Vulkan<i> pin instead of being misread as CUDA. This also keeps the Chat Settings picker reliably disabled on Vulkan builds (main.py gates gguf_gpu_ids_supported on this detection), so the UI never emits physical indices into the Vulkan-ordinal route. - Frontend: fold activeMemoryMode into loadedGpuMemoryFields so every successful load path (primary, rollback, compare-load, GGUF/non-GGUF/MTP auto-load) resets it from the LoadResponse, not just the primary path. Prevents a stale pinned mode from an earlier model being re-sent by an immediate same-model Apply. - Add test_is_vulkan_backend_matches_versioned_soname. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Pin get_device in gpu_ids GGUF route test for GPU-less CI test_inference_route_validates_gpu_ids_for_gguf patches resolve_requested_gpu_ids to exercise the CUDA physical-ID resolver, but did not pin the device. On a GPU-less CI runner get_device() returns non-CUDA, so the route took the non-CUDA/Vulkan deferral branch whose 'no GPU backend detected' 400 contains the substring 'not supported', tripping the test's assertNotIn. Pin get_device to CUDA so the test deterministically covers the path it intends. * fix(tests): simplify ordering assertions and tighten GPU validation error message * fix(studio): add GGUF variant check before reusing draft extras; apply aggregate headroom to Vulkan multi-GPU pins - inference.py /load path: check extra_args_source and gguf_variant before reusing a loaded server's extra_args for draft-device validation, so a different quant of the same repo no longer inherits stale --spec-draft-device. - inference.py /validate path: same variant-aware check before inspecting stored backend extras for draft-device rejection. - training_vram.py: Vulkan multi-GPU pin now also enforces the aggregate _MULTI_GPU_OVERHEAD (0.85) check instead of returning early on per-GPU min_free alone, preventing a chat load from starting with too little protected headroom and OOMing an active training run. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix(studio): harden Vulkan probe/budget and diffusion classification for GGUF placement - Use the versioned-soname matcher in the Vulkan free-memory probe too, matching _is_vulkan_backend: a split-library install that ships only libggml-vulkan.so.0 is now sized correctly instead of returning no devices and rejecting gpu_ids. - Budget the Vulkan multi-GPU aggregate over the least-free N pinned cards, not all visible GPUs: with the ordinal to physical mapping unknown, summing every visible card could approve a pin that no N-card placement can actually hold. - Classify a GGUF from its local header before the name heuristic: a normal llama-server GGUF whose path or repo merely contains "diffusion" is no longer rejected for gguf_memory_mode, since the loader routes on the decoded header. - Compute the Vulkan-ordinal flag before diffusion_gpu and gate diffusion_gpu off when it fires, so an unclassified GGUF pinned on a Vulkan build is budgeted as ordinals instead of the single-device CUDA path sizing the wrong physical card. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix(studio): gate XPU gpu_ids rejection to non-Vulkan; skip GGUF memory-mode reject for non-GGUF - /load and /validate paths now detect the Vulkan build before the XPU rejection so Vulkan-backed Intel GPUs can use gpu_ids via --device Vulkan<i> ordinals instead of being blocked by the torch-xpu device check. - _reject_diffusion_memory_mode now returns early for non-GGUF configs so a stale or shared payload with gguf_memory_mode cannot block unrelated non-GGUF loads. * fix(studio): skip llama.cpp GPU-lib check for diffusion; resolve versioned Vulkan libs in probe - /load and /validate paths now skip _backend_lacks_gpu_lib for confirmed DiffusionGemma GGUFs, since the diffusion runner bypasses llama-server and handles gpu_ids via --gpu/DG_GPU independently of the llama.cpp build. - _vulkan_probe.py now resolves versioned sonames (libggml-vulkan.so.0 etc.) via a directory scan so split-lib installs that lack the unversioned symlink are no longer rejected with no visible devices. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * test(studio): cover diffusion gpu_ids skip + versioned Vulkan soname; mirror is_vulkan_build on draft-device stubs - Add is_vulkan_build to three draft-device test stubs so they mirror the real backend: the gpu_ids validation now reads it before the draft-device reject. - test_cpu_only_llama_build_skips_reject_for_diffusion_gguf: a diffusion GGUF is served by the visual-server runner, so a CPU-only llama.cpp build does not reject its gpu_ids (the flow reaches the id resolver past the skipped reject). - test_versioned_only_vulkan_soname_is_probed: a split-library install shipping only libggml-vulkan.so.0 is still classified Vulkan and probed, not rejected. * fix(studio): route confirmed diffusion gpu_ids through the CUDA path on a Vulkan build A confirmed diffusion GGUF is served by the visual-server runner, which takes a CUDA physical id via --gpu/DG_GPU (it remasks CUDA_VISIBLE_DEVICES), never a Vulkan ordinal. The gpu_ids preflight previously sent it to the Vulkan-ordinal branch whenever the installed llama-server was a Vulkan build, so a valid physical pick could be rejected (no matching Vulkan ordinal) or an ordinal validated that the runner then applies to the wrong CUDA GPU. Route a confirmed diffusion GGUF through the CUDA resolver at both /load and /validate even on a Vulkan build, matching the training guard's existing diffusion_gpu handling. * Keep llama.cpp fit headroom * Adapt host memory modes to llama.cpp load mode * feat(studio): expose GGUF host memory controls * Clarify GGUF host memory semantics * Expose Vulkan ordinals in the GGUF GPU picker * Show Vulkan hardware names in the GPU picker * Clarify the llama.cpp Vulkan backend selector * Keep explicit Vulkan choices authoritative * Preserve requested and effective GGUF GPU pins * Align GGUF dedupe fixture with requested GPU pins * Restore Vulkan diffusion pin guard * Close GPU backend integration gaps * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Simplify GGUF placement state and validation * Trim GGUF placement comments * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Address GGUF placement review findings * Address GGUF placement review follow-ups * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Reject invalid remote diffusion placement before teardown * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Correct Vulkan GGUF placement boundaries * Simplify Vulkan metadata handling * Preserve deferred GPU index namespaces * Preserve deferred GPU and memory settings * Preserve host memory in compare loads * Handle DiffusionGemma memory settings * Use semantic preset summary font size * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Enforce host memory environment overrides * Keep host memory default implicit * Clarify host memory settings * Clarify host memory choices * Clarify Host RAM choices * Explain Host RAM behavior * Limit Host RAM wording change * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Guard GPU selections by backend namespace * Correct Host RAM placement semantics * Restore the GGUF fit target * Remove GGUF Host RAM controls * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Correct GPU placement comments * Fix DiffusionGemma GPU selection capabilities * Make GPU selections own main device placement * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Trim GGUF placement regression coverage * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Remove orphaned memory policy and close Vulkan gaps * Correct draft-device recovery guidance * Fix Vulkan source and training device handling * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Align automatic Vulkan device pools * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fail closed on explicit Vulkan no-space, refresh the GPU picker on probe recovery setup.sh and setup.ps1 both dispatch the prebuilt installer's exit status before checking whether Vulkan was explicitly requested, so status 4 (no disk space) took the no-space branch and never reached the strict-backend check in the else. With an older CUDA, ROCm or CPU llama-server already installed, _has_local_llama_server kept _LLAMA_CPP_DEGRADED false, so setup exited 0 and Studio carried on using the backend the user asked to replace. Driving the real dispatch block with a stubbed installer shows the same explicit request exiting 1 on a generic failure and 0 on a no-space failure. Apply the same check to the no-space path in both scripts. useGpuDevices fetched /api/system once from an effect with no dependencies and subscribed to nothing. When the Vulkan probe is unavailable, main.py answers gguf_devices as an empty list rather than omitting it, so the "?? devices" fallback does not apply and the picker starts hidden. useInferenceGpuInfo then retries every 3 seconds and refreshes the shared cache, but nothing told the mounted device hook, so the picker stayed hidden until it was remounted. Notify subscribers when a refetch replaces the cache and have useGpuDevices subscribe, comparing by value since each refresh builds a fresh array and an unconditional set would re-render on every retry tick. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix Vulkan placement state and installation * Use published Vulkan bundles * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Consolidate llama.cpp backend selector * Specify UTF-8 for file operations * fix(studio): harden GGUF GPU placement * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix(studio): align backend selection contracts * fix(studio): align MTP draft device placement * fix(studio): address Codex P2s on legacy Vulkan env var and cold-cache GPU pins - llama_backend_from_env() (and setup.sh/setup.ps1's shell-level mirror) falls back to the legacy UNSLOTH_LLAMA_BACKEND when UNSLOTH_LLAMA_CPP_BACKEND is unset, so an existing UNSLOTH_LLAMA_BACKEND=vulkan environment keeps forcing Vulkan across the setup.sh/setup.ps1 consolidation instead of silently reinstalling CUDA/ROCm/CPU. - loadedGpuMemoryFields() no longer drops a just-applied gpu_ids pin to null when cachedPinnableGpuIndexKind() returns undefined (cache cold / Vulkan probe not ready yet). That state is deferred, not rejected: the pin is kept so a reload/rollback during that window still sends gpu_ids instead of letting llama.cpp fall back to every device. The third P2 (force-compile skipping the Vulkan-source-build rejection) was already fixed by an earlier commit on this branch -- verified _NEED_LLAMA_SOURCE_BUILD is set before the guard runs in both setup.sh and setup.ps1, and added a regression test locking in the ordering. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: fix Vulkan backend env fallback and GGUF preflight token Three narrow correctness fixes on the Vulkan GPU selection path. setup.sh / setup.ps1 consulted the legacy UNSLOTH_LLAMA_BACKEND only when UNSLOTH_LLAMA_CPP_BACKEND was empty, while install_llama_prebuilt.py's llama_backend_from_env() consults it whenever the new var is not an explicit backend name. With UNSLOTH_LLAMA_CPP_BACKEND=auto beside a legacy UNSLOTH_LLAMA_BACKEND=vulkan the shell dropped the request, leaving _explicit_vulkan_backend and _explicit_vulkan_source_build false while the installer it launches still planned Vulkan, so a Vulkan install that could not be satisfied silently degraded to a CUDA/ROCm/CPU build instead of failing closed. Mirror the Python rule in both scripts; an unrecognized value with no legacy backend is still preserved so the warning can name it. The new GGUF diffusion preflight in the chat load path sent the raw stored HF token to /validate before validateModel/loadModel could run prepareHfTokenForUse. The Hub rejects an invalid Authorization header with 401 even on a public repo, so a stale saved token aborted the whole load instead of offering the existing continue-anonymously/replace-token recovery. Prepare the token the same way the compare path already does. The training guard handed can_load_chat_during_training an empty Vulkan free-VRAM map for an unclassified GGUF on a Vulkan build. That reads as no free VRAM anywhere, so every uncached remote GGUF was refused with 409 while training ran. Only an explicit pin is unbudgetable (the ordinal belongs to one namespace and neither can stand in for the other); automatic placement has no ordinal to mis-map, so it keeps the torch view. Each fix has a regression test that fails without it. * Let the diffusion runner take physical ROCm indices device_backend is a display label: _backend_label swaps CUDA for "rocm" when IS_ROCM is set, but ROCm reuses torch.cuda.* and the DiffusionGemma runner selects by the same physical index either way. Gating on "cuda" alone marked every physical ROCm device unpinnable, so the picker vanished on multi-GPU ROCm hosts and status hydration could discard a valid selection. * fix(studio): keep unavailable Vulkan inventory out of CUDA fallback, prepare autoload HF token, restore legacy hip/rocm opt-out Three P2s from the latest Codex review round on PR #7188: - toGpuDevices(): a confirmed-Vulkan inference backend with a still-cold or transiently-empty device probe no longer falls through to the torch/CUDA inventory. That fallthrough was marking physical CUDA/ROCm devices diffusionPinnable, exposing a GPU picker for DiffusionGemma that sent IDs the backend rejects outright whenever is_vulkan_build is true. Now returns no devices until the Vulkan probe actually succeeds. - loadAutoLoadCandidate(): the startup auto-load path's diffusion-classifying fetchGgufStagedMetadata probe (fired when a cached GGUF has saved gpu_ids) sent the raw stored HF token, bypassing prepareHfTokenForUse. Unlike validateModel/loadModel (which prepare internally), fetchGgufStagedMetadata does not, so an expired token could 401 even a public repo and abort the candidate before the anonymous/replace-token recovery flow got a chance to run. Now prepares the token first, mirroring performLoad's identical guard. - _normalized_llama_backend(): dropped hip/rocm entirely during the backend- selector consolidation, despite force_vulkan_requested()'s own comment ("so =hip is a real opt-out a stale UNSLOTH_FORCE_VULKAN cannot overrule") and _route_to_vulkan_prebuilt()'s ("an explicit hip/cpu is the opt-out") describing behavior it no longer delivered. An existing UNSLOTH_LLAMA_BACKEND=hip/rocm environment on an unsupported HIP arch read as "no backend named" and silently routed to Vulkan (auto-fallback, or a stale UNSLOTH_FORCE_VULKAN=1). Restored hip/rocm recognition (canonicalized to "hip"), matching the pre-consolidation mapping. Regression tests added for all three. * Cover the auto-load HF token preflight behaviourally The contract test added alongside the fix asserts on the source text. This runs the real classification block out of chat-adapter.ts under node with a stubbed Hub that 401s any non-null Authorization value, so it fails on the token value that actually reaches /api/inference/validate rather than on a symbol being present. Reverting only the source (leaving the import in place) reddens it with the simulated 401, not an ImportError. * Converge Vulkan backend selection and review fixes * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Stop the structlog stub shadowing the real package for PR #7188 The bare setdefault parked an empty placeholder before anything imported the real structlog, so every later module calling structlog.get_logger at import time raised AttributeError. It only bit when this file was collected first, which is why the 8 hardware-dispatch cases passed alone and failed under pytest tests/studio. Only stub when the package is genuinely absent. * Prepare model settings GGUF token preflight * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Keep the established llama.cpp backend selector * Preserve GPU namespace while Vulkan inventory recovers * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix final PR 7188 review findings * Normalize llama.cpp device flag aliases * Converge PR 7188 review findings * Honor backend opt-outs for Intel Vulkan routing * Retry cold system discovery for GPU selection * Probe structlog with find_spec instead of a bare import The availability check imported the module purely for its side effect, so the import-hoist verifier flagged it as an added-but-unused import and failed Source lint. find_spec answers the same question without binding a name. * Brace the PowerShell variable in the setup.ps1 backend-choice probe * Leave an existing structlog entry alone in the availability probe find_spec raises ValueError on a module already in sys.modules whose __spec__ is None, which is what a bare types.ModuleType stub is. Check sys.modules first so anything already present, real or stubbed, is left untouched and only a genuinely absent package gets stubbed. * Consolidate duplicated Vulkan selector tests without changing behaviour Three of the tests added for the llama.cpp backend selector asserted on the same two helpers with the same inputs, and two more differed only in whether the backend arrived from the environment or from --llama-backend. Fold them into the parametrized cases that already covered those helpers: - test_llama_cpp_backend_env_requests_vulkan, test_llama_cpp_backend_auto_does_not_trigger_vulkan and test_hip_backend_env_opts_out_of_vulkan become rows on test_force_vulkan_requested_accepts_public_selector_and_legacy_alias, which now also asserts llama_backend_from_env() alongside force_vulkan_requested(). The unset case and the whitespace-padded HIP/ROCM values keep their coverage as new rows. - test_explicit_non_vulkan_backend_suppresses_intel_auto_route and test_non_vulkan_backend_argument_suppresses_intel_auto_route become one test parametrized over the env and argument paths. - test_unclassified_gguf_without_pin_keeps_the_torch_budget and test_unclassified_gguf_with_pin_refuses_to_budget shared the same four patches and differed only in the pin, so they become subtests of one case. No source changes and no assertion is dropped. Each behaviour was re-checked by reverting the source hunk it guards and confirming the surviving test fails on a value. * Keep the standing GPU mode on diffusion loads and the ARM64 Vulkan fallback for PR #7188 * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Make the Intel auto-route guard test able to fail and stop the ps1 harness doubling --force-cpu test_non_vulkan_backend_suppresses_intel_auto_route ran on Linux x86_64, where dropping the explicit_backend guard changes only a log line, so the test passed either way. Move it to Linux ARM64 + Intel GPU, the host where the guard decides the routed repo, and assert that value. _run_ps1 composed the --force-cpu snippet twice: it is already inside the normalized block the helper extracts. The substring assertion hid the duplicate. Compose it once and compare the whole argv. setup.ps1 appends the flag once and is unchanged. * Dot-source Get-HostMachineArch in the setup.ps1 Pester suite #7549 taught Test-VCRedistInstalled to consult the host architecture before trusting the System32 DLL. The Pester suite dot-sources a fixed list of functions out of setup.ps1 one at a time, and that list did not gain the helper, so the two clean-box cases (registry miss, and an old sub-14.20 redist) throw "Get-HostMachineArch is not recognized" instead of returning false. The registry hit returns early, which is why the other cases pass. #7597 made exactly this fix to the sibling list in the VC++ round-trip job but left the Pester suite alone. Verified with pwsh 7.6 + Pester: 3 failed before, 31 passed 0 failed after. * Drop the branch's Pester Get-HostMachineArch fix now that #7606 landed This branch carried a local fix for the same Pester dot-source gap that #7606 has since merged to main (f21b280). Keeping both leaves the PR diff with an unrelated, now-duplicated change, so drop the branch copy and take main's. --------- Co-authored-by: danielhanchen <unslothshared@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: oobabooga <112222186+oobabooga@users.noreply.github.com> Co-authored-by: alkinun <alkinunl@gmail.com> Co-authored-by: Daniel Han <danielhanchen@gmail.com> Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com> Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> Co-authored-by: danielhanchen <unslothai@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Studio currently decides how a GGUF model is placed on the GPU(s), with no way to override it: you cannot choose which GPUs to use, how many layers go on the GPU, or how MoE experts are offloaded. This PR adds those controls.
Closes #4870
Closes #5025
Closes #6320
Closes #6530
Controls
The main addition is a GPU Memory dropdown:
--fit on. Studio floors an automatically sized context at 8192 tokens and targets a 512 MiB margin per GPU. Fitting prioritizes dense tensors, so overflow including MoE experts can spill to RAM. If you select a context length,--fitkeeps that length and adjusts placement around it.--fit offand is attempted as requested. Studio does not resize the context or layer count to rescue a configuration that does not fit.--tensor-split: the values are layer counts in layer-split mode and relative shares under Tensor Parallelism.A GPUs checkbox group chooses which GPUs llama.cpp may use. Leaving every GPU checked keeps the automatic device set. The picker is hidden when Studio cannot map the reported devices to physical GPU indices.
Tensor Parallelism remains available in Default mode and for explicit Manual placement with at least two selected GPUs. It is hidden with GPU Layers on Auto because llama.cpp's
--fitis incompatible with tensor split mode.The existing Remember settings option now stores GPU Memory mode, GPU Layers, MoE offload, and the GPU selection per GGUF quant, alongside the other load settings. The per-GPU split is deliberately recomputed for the current GPU set rather than persisted.
Behavior boundaries
--fit; explicit layers are loaded as requested, and a configuration that does not load remains the user's responsibility.Verification
Local verification passes: 161 targeted backend tests covering GPU memory mode, active-training loads, GGUF reload inheritance, and tensor-parallel vision behavior; Ruff on the changed backend files; targeted frontend ESLint; and the frontend production build.