VMware→KVM: cold (VDDK) and warm (CBT) migration into Ceph/RBD and Linstor, plus importVm adoption of existing RBD root volumes and Linstor root/data volumes - #13656
Conversation
Operator-driven warm VMware-to-KVM migration built on VDDK and VMware Changed Block Tracking (CBT). Keeps a source-equivalent KVM-side replica current through delta cycles, then finalizes with virt-v2v in-place at cutover. Supports both filesystem (qcow2) and native Ceph/RBD (raw image) primary storage targets, with preflight validation, async lifecycle operations, persisted migration state, and Import/Export UI integration. Signed-off-by: Andrija Panic <andrija.panic@gmail.com>
Extends the existing VDDK + virt-v2v VMware-to-KVM import path so the converted disk can be written straight into an RBD pool as a raw image, with in-place virt-v2v finalization. Adds host-capability detection for qemu-img RBD support, RBD qemu copy, virt-v2v in-place, and direct RBD import, advertised via ReadyCommand and reconciled by the agent manager. Signed-off-by: Andrija Panic <andrija.panic@gmail.com>
Extends the KVM unmanaged import / manage-volume flow so an existing raw RBD image can be adopted directly as a VM root volume, including the RBD-format check in the volume orchestrator and the libvirt volume inspection path. Signed-off-by: Andrija Panic <andrija.panic@gmail.com>
Unifies the host-capability keys that the CBT warm-migration and VDDK direct-RBD-import features introduced independently under different names, using feature-neutral names that describe what is probed: host.vmware.cbt.support -> host.vddk.blockcopy.support host.vmware.cbt.in.place.finalization.support -> host.vddk.blockcopy.inplace.finalization.support host.vmware.cbt.rbd.support -> host.vddk.blockcopy.rbd.support host.qemu.img.rbd.support -> host.qemu.rbd.support host.virt.v2v.inplace.support / host.virtv2v.in.place.version -> host.virtv2v.inplace.* Drops the redundant host.rbd.qemu.copy.support alias (identical to host.qemu.rbd.support) and its hostSupportsRbdQemuCopy() delegate. Signed-off-by: Andrija Panic <andrija.panic@gmail.com>
andrijapanicsb
changed the title
…isk import The "Manage Instances" source dropdown offered "Import QCOW2 image from Local/Shared Storage", but importVm now adopts an existing ROOT disk from Filesystem, NetworkFilesystem, SharedMountPoint and RBD (raw) pools, so the QCOW2-only wording is misleading. Rename both entries and their wizard title/description to "Import Instance using existing ROOT disk from Local/Shared Storage", across the en, pt_BR and te locales. Signed-off-by: Andrija Panic <andrija.panic@gmail.com>
docs/vmware-cbt/architecture.md and docs/vmware-cbt/README.md were added without the Apache license header, which failed the RAT license check and the pre-commit insert-license hook. Add the standard ASF header (identical to the one already present in docs/vmware-cbt-migration.md). Signed-off-by: Andrija Panic <andrija.panic@gmail.com>
…ey names State Apache CloudStack 4.24.0.0 as the target release (schema delivered via the 4.24.0.0 upgrade path, with a note that the DDL currently rides in schema-42210to42300.sql pending the main version bump), and replace the pre-consolidation host-capability key names with the current host.vddk.blockcopy.* / host.virtv2v.inplace.version names. Signed-off-by: Andrija Panic <andrija.panic@gmail.com>
Staged VMware-to-KVM imports (virt-v2v conversion to an NFS temporary location followed by a copy to the destination pool) can now target Linstor primary storage: - Linstor added to the staged-conversion destination pool types, so destination pool discovery considers cluster- and zone-wide Linstor pools alongside NFS and RBD. - The import host is validated to have access to each Linstor destination pool (it must be a LINSTOR satellite connected to the pool); the RBD qemu-copy capability check is preserved and the validation renamed to validateStagedImportHostSupport. - The KVM-side converted-disk move already dispatches by pool type to LinstorStorageAdaptor.copyPhysicalDisk, which spawns the resource from the pool's resource group and qemu-img converts the qcow2 into the DRBD block device; converted-disk metadata for Linstor pools now reports the pool UUID and volume name instead of parsing NFS mounts (Linstor pools have no local mount path). The imported volume keeps the copy-time UUID as its path; the Linstor driver addresses existing volumes by path, so later volume lifecycle operations (attach, resize, snapshot, delete) work unchanged.
Extends the direct (non-staged) VDDK import path, previously available only for Ceph/RBD, to Linstor primary storage. With usevddk=true and forceconverttopool=true on a Linstor conversion pool, the KVM host copies each VMware disk over nbdkit/VDDK straight into a pre-created DRBD block device and finalizes it with virt-v2v in place - no NFS staging and no intermediate qcow2 copy. Server side: - Linstor added to the VDDK direct-conversion pool allow-list; the RBD/Linstor "requires usevddk" validation is shared. - Conversion host selection for Linstor requires VDDK support, in-place virt-v2v support, and access to the Linstor pool (the host must be a LINSTOR satellite); explicit host selections are validated for pool access as well. - CheckConvertInstanceCommand carries a new in-place finalization check so unsupported hosts fail fast with a clear message. - The powered-off, non-cloned source VM requirement now applies to any direct block-storage import. KVM agent side: - The direct conversion routine handles both targets: for Linstor it pre-creates each resource at the source capacity through the storage adaptor (qemu-img cannot create DRBD devices, unlike RBD images), copies with qemu-img convert -n to the local DRBD device, and feeds virt-v2v-in-place a <disk type='block'> domain XML - no qemu URI or RBD-style network disks needed. - Storage access is probed with a 4 MiB test volume and a qemu-io write/read round-trip before any data transfer. - Linstor disk names use a shorter '-dNN' suffix: LINSTOR resource names are capped at 48 characters, which the RBD '-disk-NNN' scheme would exceed. - The forced-conversion import short-circuit accepts Linstor temporary locations, backed by a new LinstorStorageAdaptor.listPhysicalDisks implementation (resource definitions filtered by the pool's resource group), replacing the previous UnsupportedOperationException.
Extends the VMware CBT warm-migration framework, previously limited to qcow2 file and Ceph/RBD targets, with a generic raw-block-device target type backed by Linstor primary storage. The initial full sync, the incremental CBT delta cycles and the powered-off cutover all write straight into the DRBD block device of a pre-created Linstor volume. Target classification and orchestration: - New VmwareCbtTargetStorageType.RAW_BLOCK_DEVICE; Linstor pools classify as supported block-device targets that require in-place virt-v2v finalization (the qcow2 fallback cannot write to devices). - Linstor added to the CBT-compatible pool types for explicit and implicit destination pool selection. - Conversion host selection requires VMware CBT migration support, in-place finalization support and access to the Linstor pool (the host must be a LINSTOR satellite); both auto-selection filtering and explicit host validation enforce it. - The destination storage probe now covers block-device pools: the agent creates a 4 MiB volume through the storage adaptor, round-trips a qemu-io write/read on the local device and deletes it. Naming: block-device target volumes are named cbt-<mig8>-<diskId>, where mig8 is the first eight characters of the migration UUID. LINSTOR resource names are limited to 48 characters (the volume name gets a cs- prefix as a LINSTOR resource), so the RBD-style names carrying the full migration UUID do not fit. The short marker keeps cleanup guarded: both the management server and the agent only delete volumes whose names start with the marker derived from the migration UUID. Agent-side data path: - Initial sync pre-creates each Linstor volume at source capacity via the storage adaptor (qemu-img cannot create DRBD devices) and runs the nbdkit/VDDK captive copy with qemu-img convert -n to the local device path. - Delta cycles resolve the device path and patch changed extents with the existing qemu-io write mechanism; raw device targets are accepted by disk validation. - Cutover feeds virt-v2v-in-place a <disk type='block'> domain XML with the local device path - the qemu-nbd localhost bridges required for RBD are unnecessary for block devices. - Cleanup deletes only marker-guarded volumes through the adaptor. The finalized volumes are imported through the existing shared path; imported Linstor volumes keep the target name as their volume path, which the Linstor driver uses to address existing volumes.
Covers the release note for the three import modes (staged, direct VDDK, CBT warm migration), the operator guide's destination-specific host requirements and operational notes (satellite/node naming, single Primary writer, diskful replica preference, DRBD size rounding, heartbeat fencing interaction, retained volume names), and the as-built design guide's storage-target section (RAW_BLOCK_DEVICE mapping, pre-created devices, convert -n, block-disk XML finalization, 48-character resource-name constraint and the short cleanup marker).
Found during live validation on an Ubuntu 24.04 LINSTOR lab node: virt-v2v-in-place 2.4.0 does not know the -O (write updated output XML) option, which only exists from virt-v2v 2.5 on (EL9 ships 2.5.x, where the RBD flows were originally validated). Both the VMware CBT cutover finalization and the direct VDDK import finalization passed -O unconditionally, so otherwise fully capable hosts failed the in-place step at the very end of a migration. Nothing ever consumed the output XML - both wrappers created a temporary file for it and deleted it unread - so the option and the temporary files are removed rather than gated on a version probe. Verified live: virt-v2v-in-place 2.4.0 with the same libvirt XML converts a guest on a DRBD block device successfully without -O.
…s can be deleted cancelVmwareCbtMigration already removes the migration's target images from the destination pool, but it leaves the migration record in place. Deleting the record afterwards runs the cleanup a second time, because deleteVmwareCbtMigration defaults cleanup to true, and that second pass failed on targets which were legitimately already gone. The result was that cancelled migration records could never be removed, even though no storage was left behind. deletePhysicalDiskWithRetries now treats a target as successfully deleted when the destination pool listing confirms it is no longer present. Absence is only reported when it can be positively confirmed; if the pool cannot be inspected the target is assumed to still exist, so genuine cleanup failures are still surfaced rather than masked. This applies to both the RBD and the Linstor cleanup paths, and also covers failed migrations whose cleanup only partially completed.
Making cleanup idempotent changed the contract of the cleanup wrapper: a target the destination pool no longer lists is now treated as successfully removed. The existing failure test stubbed only deletePhysicalDisk and left listPhysicalDisks unstubbed, so the pool appeared empty and the target read as already absent, which is not what that scenario describes. The test now states its premise explicitly - the image is still listed on the pool, so the deletion genuinely failed and cleanup must report a failure. Two further tests pin the rest of the contract: a target the pool no longer lists is reported as cleaned up, and a pool that cannot be inspected still yields a failure, so absence is never merely assumed.
|
@blueorangutan package kvm |
|
@andrijapanicsb a [SL] Jenkins job has been kicked to build packages. It will be bundled with kvm SystemVM template(s). I'll keep you posted as I make progress. |
Functional test report — VMware to KVM cold/warm migration, RBD and Linstor destinationsFull end-to-end functional test of this PR, run on a two-host KVM cluster against three destination Result: 15 of 15 planned test cells pass. All cells were run from scratch on the final build. Source VMs were migrated directly. No clones or Guests used: Rocky Linux 9.8 (2 disks), Ubuntu 26.04 LTS (1 disk), Windows Server 2012 R2 (2 disks). How results were verifiedA VM was only recorded as booted when the guest itself provided the evidence. CloudStack reporting
Data integrity methodEach source data disk carries a test file ( After cutover, each migrated VM was booted and the checksums were computed again, from inside the Summary
Disk orderingThe ordering behaviour is correct on every destination type. In all dual-disk cells the result was The single-disk source produced exactly one volume, with no additional data disk created. This was Warm CBT: delta cycles transfer only changed blocksThis was measured rather than assumed. The full disk set per VM is 20 GB root plus 5 GB data, i.e.
The Linstor rows come from a single migration and vary only the amount written into the running guest: Two results follow from this. An idle cycle moving 2.9 MiB out of 25 GiB is not a full re-copy, and CloudStack reports this per cycle, for example:
together with a Data path coveragenbdcopy path and qemu-img fallback, with log outputWith and Linstor writes directly to the DRBD device: For cell C5, The resulting VM booted and the checksum taken inside the guest matched the source value, so the Adoption of a pre-existing volumeA standalone bootable RBD image was created outside CloudStack and then adopted with: The resulting ROOT volume's What the import covers, and what it does not. On the shared/local disk path, So if the VM has more than one disk, or more than one network, you need extra API calls after the
This matters when migrating a VM with several disks and NICs: the import gets you the root disk and one Negative testing casesAll rejections were returned quickly, and each message identifies both the problem and the required
N4 and N5 are both validated at start, within seconds, rather than after a full replication or a large N3 (cancel) was checked by inspecting the storage rather than relying on the API result. During Windows guestsScope for the Windows cells was boot verification. The original Windows Server 2012 R2 source was Cold import (X1)
Warm CBT migration (X2) Change tracking was enabled on the source, the initial full sync completed with the source powered on, In-guest networking and data contents were out of scope for the Windows cells. NotesDisk controller on adopted volumes. Adopting a volume without further parameters produced Disks are copied one at a time. On both the cold and warm paths, disks are converted sequentially Scope of this testing
|
|
Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✖️ debian ✔️ suse15. SL-JID 18725 |
|
@blueorangutan test |
|
@andrijapanicsb a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests |
|
@blueorangutan test rocky9 kvm-rocky9 keepEnv |
|
@andrijapanicsb a [SL] Trillian-Jenkins test job (rocky9 mgmt + kvm-rocky9) has been kicked to run smoke tests |
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #13656 +/- ##
============================================
+ Coverage 19.64% 19.77% +0.13%
- Complexity 19789 20204 +415
============================================
Files 6368 6420 +52
Lines 575109 582046 +6937
Branches 70371 71206 +835
============================================
+ Hits 112994 115115 +2121
- Misses 449829 454446 +4617
- Partials 12286 12485 +199
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
5 participants
Description
This PR makes Ceph/RBD and Linstor first-class targets for KVM VM onboarding and adds an operator-driven warm VMware→KVM migration path. It bundles the following (Ceph/RBD plus Linstor as block-storage destinations):
1. VMware CBT → native RBD/qcow2 warm migration to KVM
An operator-driven, warm VMware-to-KVM migration built on VDDK and VMware Changed Block Tracking (CBT). It maintains a source-equivalent KVM-side replica kept current through repeated delta/sync cycles, then finalizes the guest with virt-v2v in-place at cutover — bounding downtime to the cutover window instead of a full cold copy.
qcow2) and native Ceph/RBD (raw image) primary storage.startVmwareCbtMigration,syncVmwareCbtMigration,cutoverVmwareCbtMigration,cancelVmwareCbtMigration,deleteVmwareCbtMigration,listVmwareCbtMigrations,checkVmwareCbtMigrationPrerequisites.VmwareCbtMigrationManagerImpl(+VmwareCbtMigrationServiceImpl, cutover policy), backed by new VOs/DAOs and a DB schema migration.docs/vmware-cbt/(architecture + README) anddocs/vmware-cbt-migration.md.2. Direct VDDK VMware import into Ceph/RBD — the cold (one-shot) path
The cold counterpart to the warm CBT migration in (1): extends the existing
VDDK + virt-v2v VMware→KVM import — which previously landed only on NFS/
qcow2(direct, or via a staged file) — so the converted disk is written straight into an
RBD pool as a raw image, with in-place virt-v2v finalization and no intermediate
qcow2-on-NFS staging copy. (The warm path in (1) reuses this same full-disk copy for its
initial sync.) Adds host-capability detection for qemu-img RBD support, RBD qemu-copy,
virt-v2v in-place, and direct-RBD import — advertised via
ReadyCommandand reconciledby the agent manager.
3.
importVmadoption of existing RBD (Ceph) root volumes and Linstor root + data volumesExtends the KVM unmanaged-import / manage-volume flow so an existing raw RBD image can be adopted directly as a VM ROOT volume. Adds RBD-aware format handling in
VolumeOrchestrator(RBD ⇒RAWinstead of the hardcodedQCOW2) and the RBD branch in the libvirt volume-inspection path (LibvirtCheckVolumeCommandWrapper), soCheckVolumeCommandsucceeds on RBD instead of failing with "Disk not found or is invalid".Also extends the same adoption path to Linstor (DRBD) primary storage, for both ROOT and DATA volumes: an externally prepared Linstor volume (e.g. landed by a migration/conversion tool) can be adopted without any data copy. Linstor volumes are raw local block devices, so volume inspection runs qemu-img against the
/dev/drbd/…device path (LibvirtCheckVolumeCommandWrapperfor ROOT,LibvirtGetVolumesOnStorageCommandWrapper+ the volume import/unmanage gate for DATA), andVolumeOrchestratorrecordsRAW. Because a host-local qemu-img file lock cannot see a DRBD resource that is Primary/open on another node (unlike RBD's cluster-wide exclusive lock), adoption additionally consults Linstor's cluster-wideInUsestate — newKVMStoragePool.getVolumeInUseNode()capability (default no-op), answered via the existingLinstorUtil.isResourceInUse— and refuses a volume attached to a running VM anywhere in the cluster. Volumes are adopted under their existing resource names (Linstor cannot rename; 48-char limit).The existing filesystem/
qcow2adoption path is left unchanged, by construction rather than by accident: the supported-pool-type list is extended additively; the raw-device inspection branch is guarded by an explicit RBD/Linstor check that returns early, so filesystem pools still fall through to the originalcheckQcow2Filepath; the new cluster-wide in-use lookup is aKVMStoragePooldefault returningnulland is overridden only byLinstorStoragePool; and the format resolution is a new overload that delegates to the pre-existing method for every pool type other than RBD/Linstor (soqcow2pools still resolve toQCOW2).4. Consolidate KVM host-capability probe names across the RBD features
Because features (1) and (2) introduced host-capability keys independently, this commit unifies them under feature-neutral names that describe what is probed, and drops a redundant alias (
host.rbd.qemu.copy.support, identical tohost.qemu.rbd.support) and its delegate.5. Linstor (DRBD) primary storage as a migration destination
Extends all three VMware→KVM paths above — staged import, direct VDDK import, and warm CBT migration — so the converted disks can also land on Linstor primary storage, not just Ceph/RBD. A generic
RAW_BLOCK_DEVICEtarget type covers "write RAW into a host-local block device provided by the storage adaptor", so the same code paths serve Linstor (and lay groundwork for other block backends).qemu-imgcannot create a DRBD device), copies withqemu-img convert -n— ornbdcopywhen the host has it — into the local/dev/drbd/…device, and finalizes withvirt-v2v-in-placefed a<disk type='block'>domain XML (no qemu-nbd bridge needed for a local device).qemu-img convert -n -S 0), avoiding a temp-file round-trip.-Oflag from thevirt-v2v-in-placeinvocation (fixes in-place finalization on virt-v2v < 2.5, e.g. Ubuntu 24.04) — which benefits the RBD path too.6. Data-plane efficiency and correctness for the agent-driven copies
These apply to the agent's own nbdkit/VDDK full-disk copies (cold direct import and CBT initial full sync); they do not touch the staged virt-v2v conversion path.
compression=<value>(newvddk.nbd.compressionagent property, defaultfastlz;none/zlib/fastlz/skipz). On datastores that don't expose block allocation to VDDK, the full nominal disk size otherwise crosses the network;fastlzcompresses both zero regions and real data on the ESXi side at low CPU cost (measured on a 30 GiB disk: 32.4 GB → 2.65 GB over the wire).nbdcopyfor RBD full copies — cold-direct-to-RBD and the CBT initial full sync to RBD now copy withnbdcopyover a localhostqemu-nbdbridge (pre-create the raw image, serve it locally, copy NBD-to-NBD) instead of a single-streamqemu-img convert, falling back toqemu-img convertwhennbdcopyis unavailable. The Linstor block-device path already usednbdcopydirectly; CBT delta cycles are unchanged (a range copier is required there).--target-is-zero/--destination-is-zero) only when the backend guarantees a freshly created volume reads back as zeros, via a newKVMStoragePool.isVolumeZeroInitialized()capability (Linstor answers it from the provider kind — thin providers only). On backends that do not guarantee it (e.g. LVM-thick) the zeros are written, so no stale data from a previously deleted volume leaks into the unwritten regions.qemu-img's--target-is-zerosupport is also probed directly (newer qemu dropped it from--helpwhile still supporting it).7. Windows guest post-migration handling
For Windows guests, a first-boot script is injected via
virt-v2v --firstboot: on first boot it sets the SAN policy toOnlineAlland brings migrated data disks online and read-write, closing the common post-conversion gap where secondary disks come up Offline/Read-only under the default Windows SAN policy. It is best-effort and fail-safe — if it cannot run, the conversion still succeeds and the disks simply keep their default state. (Uses Storage cmdlets available on Windows Server 2012+.)Networking / static-IP scope. After conversion a Windows guest gets a new virtio NIC that comes up with its default (DHCP) configuration. On a CloudStack-managed network this is the intended, deterministic outcome — the source MAC is preserved and the virtual router hands the guest exactly the IP CloudStack allocated for that NIC (a DHCP reservation keyed on the MAC), so no extra action is needed. Automatically re-applying an in-guest static IP (for DHCP-less networks, or guests that must remain internally static) is intentionally out of scope here — doing it reliably needs guest-registry inspection to distinguish static from DHCP and recover prefix/gateway/DNS, which is better suited to a dedicated follow-up. For those cases the operator supplies the address via
nicipaddresslistand, on a DHCP-less destination, reconfigures networking inside the guest after migration.Types of changes
Feature/Enhancement Scale
How Has This Been Tested?
Unit tests for all new and touched wrappers/managers/orchestration:
LibvirtImportConvertedInstanceCommandWrapperTest,LibvirtConvertInstanceCommandWrapperTest,LibvirtCheckConvertInstanceCommandWrapperTest,LibvirtCheckVolumeCommandWrapperTest,the
LibvirtVmwareCbt*suites,UnmanagedVMsManagerImplTest,VolumeImportUnmanageManagerImplTest, the server-sideVmwareCbt*suites,and
VolumeOrchestratorTest— all passing.Manual/lab validation of warm CBT migration and direct-RBD import against a Ceph/RBD KVM cluster.
Linstor destination validated byte-identical on a live 3-node LINSTOR/DRBD lab against a real VMware source (cold direct-VDDK and warm CBT;
qemu-img compare= images identical), and end-to-end through a liveimportVmthat produced a Stopped CloudStack VM with its root and data disks as Linstor volumes backed by DRBD resources UpToDate on all three nodes. New Linstor unit tests across the destination gate lists, host selection, the convert/import/CBT wrappers,listPhysicalDisks, and the block-device delta/nbdcopypaths — all passing.Linstor adoption (root + data) covered by new unit tests: CheckVolume inspection via the
/dev/drbddevice path (never anrbd:URI), the in-use-on-another-node refusal, and the Linstor ⇒RAWformat resolution — all passing; and validated live end-to-end on the 3-node LINSTOR lab: an externally-prepared Linstor root + data set adopted throughimportVmand the resulting VM started (adopt-and-boot), with the volumes backed by DRBD resources UpToDate on all three nodes.Full functional pass on a two-host KVM cluster against all three destination types (NFS, Ceph/RBD, Linstor): cold direct-VDDK imports (dual-disk, single-disk, and with
nbdcopyremoved to exercise theqemu-imgfallback), warm CBT migrations on each destination, volume adoption, native VM lifecycle, and negative/guard-rail cases. Boot was confirmed from the guest console rather than from CloudStack VM state, and data integrity by runningsha256suminside each migrated guest against checksums recorded at the source — including files written into the running source between delta cycles, which confirms the delta path carries mid-replication writes. Delta cycles were also measured: against a 25 GiB disk set an idle cycle moved ~2.9 MiB (0.011 %) and cycles tracked the volume actually written (write 96 MiB → ~101 MiB transferred), confirming changed-block transfer rather than full re-copy. Windows Server 2012 R2 was migrated cold and warm and booted in both cases, with the first-boot disk-online script confirmed firing.