Achieving fusion—with a service training doc, Ars tears open Apple’s Fusion Drive

Apple’s new Fusion Drive technology is a bit of a mystery. Dismissed by some as just another simple cache mechanism, it’s drawn a lot of attention in the weeks since it was announced. (And it wasn’t until the end of last week that Macs equipped with Fusion Drives actually began shipping.) Ars had its hands on a Fusion Drive-equipped Mac Mini for a few days now, and we set ourselves to the challenge of solving the mystery. We spent several days elbows-deep in the technology, attempting to determine how Fusion Drive (henceforth referred to as FD) works and what effect it has on overall system performance. We also wanted to see what happens with installing Windows through Boot Camp, and what happens when Fusion Drive fails. We’ve learned a lot in a short time through observation and deduction, and while some questions are still unanswered, we’ve gotten a lot of outstanding, exclusive info.

To briefly review, FD is a feature announced by Phil Schiller on the October 23 Apple event, debuting in the new Mac Minis and iMacs also announced at the event. FD uses Core Storage to create a single volume out of a solid state disk and a much larger traditional hard disk drive, and then tiers data back and forth between the two. The goal is to deliver the speed of an SSD, while also taking advantage of the large size of the HDD to hold the bulk of your data. In typical Apple fashion, though, the presentation was high on hyperbole but low on actual technical meat. We’ve done two more peeks at FD, one diving into the impressive hackery done by Patrick Stein (a.k.a. “Jollyjinx”), who discovered that FD appears to work on any Mac running 10.8.2, not just new Minis and iMacs. The other detailed preliminary hands-on results with our FD-equipped Mac Mini.

Ars also got ahold of Apple’s service technician training document for Fusion Drive, thanks to a helpful anonymous source. The document sheds more light on the technology, confirming several things we speculated about. In particular, it notes late 2012 21″ and 27.5″ iMacs come with flash “storage modules,” similar to the ones already seen on MacBook Airs and Retina MacBook Pros. Late 2012 Mac Minis use a 2.5″ SSD instead of a storage module.

The training still makes reference to FD moving “apps and documents” between SSD and HDD, though testing has shown this isn’t quite true. FD operates below the files, actually moving Core Storage blocks (which seem to be referred to internally as “chunks,” as we’ll see). Apple’s presentation of the technology as moving “apps and documents” makes perfect sense, though: apps and documents are easy to explain in a quick launch presentation, while an actual discussion of files versus file system clusters versus Core Storage blocks versus disk blocks would be difficult to cram into a single slide.

We’ve been poking and prodding at a Fusion Drive Mac Mini all weekend, and we’ve dug up lots of additional juicy information on FD’s real-world functioning. We also dove into what happens when things go wrong.

A glass half-full

Commenters in the other articles—particularly those who only skimmed the texts—have wondered at length why we’re spending so much (virtual) ink covering Fusion Drive. Isn’t it just a plain caching solution? Isn’t it the same as Intel SRT? Hasn’t Linux been doing this since 1937?

No, no, and no. Intel’s Smart Response Technology is a feature available on its newer Ivy Bridge chipsets, and it allows the use of a SSD (up to 64GB in size) as a write-back or write-through cache for the computer’s hard drive. One significant difference between FD and a caching technology like Intel SRT is that Fusion Drive alters the canonical location of the data it tiers, moving it (copying it, really, because we don’t see a “delete” file system call during Fusion migrations, as we’ll demonstrate in a bit) from SSD to HDD. More importantly, with FD, as much data as possible goes to the SSD first, with data spilling off of the SSD onto the HDD. Picture Fusion Drive’s SSD like a small drinking glass, and the HDD is a much larger bucket, below the SSD. When you put data onto a Fusion Drive, it’s like you’re pouring water into the glass; eventually, as the glass fills, water slops over the side and begins to be caught by the bucket. With Fusion Drive, you always pour into the glass and it spills into the bucket as needed.

On the other hand, caching solutions like SRT algorithmically determine what things should be mirrored up from HDD onto SSD. Even though the SSD can be used as a write cache, the default location of data is on the HDD, not the SSD. In caching, the HDD is the storage device with which you interact, and the SSD is used to augment the speed of the HDD. In Fusion Drive, the SSD is the device with which you interact and the HDD is used to augment the capacity of the SSD.

I’m definitely not going all starry-eyed over Fusion Drive, and it’s not a revolutionary new thing that will make your computer shoot rainbows out of its USB slots while curing cancer and making sick children well again. However, as we’ll see, Fusion Drive is a transparent tiering technology that simply works. It’s that seamless always-on functionality that makes it newsworthy—you buy a computer with Fusion Drive enabled and you don’t need to install or configure any additional hardware or software in order to enjoy its benefits.

Mr. Fusion

To verify what Apple says in its training, and because we just can’t leave well enough alone, we took screwdriver and spudger to our perfectly good Mini and broke it down into its component parts. We wanted to see for ourselves exactly what kind of SSD is in there.

Late 2012 Mac Mini, gutted.

Credit: Lee Hutchinson

The Mini’s SSD, a SATA III Samsung 830-based OEM model.

Credit: Lee Hutchinson

It’s a Samsung 830-based MZ-5PC1280, reportedly the same type of SATA III SSD shipping with current non-retina Macbook Pros, though apparently with updated firmware. It comes mounted in the Mini’s lower HDD bay, connected to the L-shaped motherboard with an adapter.

SATA adapter, connecting the SSD’s power and data leads to the L-shaped motherboard (or logic board, in Apple parlance).

Credit: Lee Hutchinson

Though there are indeed two physical disks, FD unifies them into a single volume. From a user’s perspective, it’s a single disk, and you interact with FD volume in exactly the same fashion as you would a standard disk. No configuration or fiddling is required.

Performance is right where you’d expect it to be for a mid-tier SSD, with XBench reporting numbers that are good, but not outstanding. Tellingly, there’s no way to bench the HDD—the benchmark applies itself to the disk subsystem, and Core Storage ensures that it interfaces with the SSD.

To actually see the SSD and the HDD separately, you have to go a little lower and pull in the command line diskutil command, which shows the Mini’s SSD and HDD are definitely two separate physical devices:

LeeHs-Mac-mini:~ leeh$ diskutil cs list
	CoreStorage logical volume groups (1 found)
	|
	+-- Logical Volume Group 31CBA650-4ABC-4DA6-AA73-6D96B36693E2
		=====================================================
		Name:         Macintosh HD
		Size:         1120333979648 B (1.1 TB)
		Free Space:   0 B (0 B)
		|
		+-< Physical Volume F4A50F9C-841D-4C6D-AFC4-6A10560E0717
		|   ----------------------------------------------------
		|   Index:    0
		|   Disk:     disk0s2
		|   Status:   Online
		|   Size:     120988852224 B (121.0 GB)
		|
		+-< Physical Volume C86789FD-1978-453A-BCA7-81DC05AA66C7
		|   ----------------------------------------------------
		|   Index:    1
		|   Disk:     disk1s2
		|   Status:   Online
		|   Size:     999345127424 B (999.3 GB)
		|
		+-> Logical Volume Family AA92FDF8-F962-4DE3-AC2F-1B6ABF22AA22
			------------------------------------------------------
			Encryption Status:       Unlocked
			Encryption Type:         None
			Conversion Status:       NoConversion
			Conversion Direction:    -none-
			Has Encrypted Extents:   No
			Fully Secure:            No
			Passphrase Required:     No
			|
			+-> Logical Volume B97C9558-14FC-4BFC-B413-AFB292BD29F0
				-----------------------------------------------
				Disk:               disk2
				Status:             Online
				Size (Total):       1115853029376 B (1.1 TB)
				Size (Converted):   -none-
				Revertible:         No
				LV Name:            Macintosh HD
				Volume Name:        Macintosh HD
				Content Hint:       Apple_HFS

If you’re a casual user and don’t care about the internals, there is nothing that you need to do to make FD “work.” You power on the system, log on, and use it. A Fusion Drive-equipped Mac leaves the factory with the operating system and all of the pre-installed applications on the SSD side, so the system is just as snappy and responsive as if it were an SSD-only Mac. Boot times out of the box on the Mini were impressively quick, with the system going from button-press to log-on prompt in about seven seconds by my stopwatch. Apple’s documentation indicates that FD will always keep the operating system and other critical files pinned to the SSD, so this snappiness should persist throughout the Mac’s life.

Digging deeper

That’s all fine and good for a brand new system, but we wanted to know how FD handles things during normal usage. One of the first things many folks are going to do to a new Mac is migrate their existing data onto it, and so we attempted to find out how FD behaves when something like that happens. Initial reports from Anandtech indicated that FD keeps a 4GB “landing zone” on the SSD, and that all writes were directed to that landing zone. This doesn’t appear to be correct. Or, rather, it appears to be partially right, but not entirely so.

Out of the box, the system showed only about 10GB in use. I copied a 3GB file from my NAS down onto the Mini while monitoring the physical drives’ throughput with iostat, and all IO occurred on the SSD. Once the copy was completed, I let the Mini sit for about an hour to see if it would move some or all of the file down to HDD, but no move occurred. I copied a second 3GB file and observed the same behavior—the copy landed on the SSD, and seemed to stay there. I followed that up with a large 8GB file, which also landed entirely on the SSD and stayed there. Throughout all of this, iostat showed no activity at all on the HDD.

The 4GB “landing zone” didn’t make itself apparent anywhere in the results, so I then decided to emulate a typical “user moving in” type of behavior by filling up the SSD with stuff to see what FD would do. I copied down about 120 GB of files, which would more than fill the SSD and force the system to show me its contingency behavior.

When the copy started, all data went to the SSD, just as before. When the amount of data on the SSD hit about 110 GB (roughly the size of the Core Storage partition on the SSD), IO shifted smoothly from the SSD (disk0 in the iostatoutput below) to the HDD (disk1), without interrupting the copy:

	   disk0               disk1 
   KB/t tps  MB/s      KB/t tps  MB/s 
 915.91 141 125.98     0.00   0  0.00 
1016.09 129 127.88     0.00   0  0.00 
1014.97 113 111.93     0.00   0  0.00 
1011.56  82 80.92      0.00   0  0.00 
1016.03 128 126.84     0.00   0  0.00 
1016.09 129 127.84     0.00   0  0.00 
1013.48  97 95.88      0.00   0  0.00 
1024.00  96 95.96      0.00   0  0.00 
 958.58  48 44.91   1007.30  57 56.04 
   4.00   1  0.00   1010.16  74 72.92 
   8.00  24  0.19   1024.00  66 65.92 
   4.00   2  0.01   1010.53  76 74.95 
  15.43   7  0.11    974.74  70 66.59 
   4.00   1  0.00    912.00  82 72.95 
   4.00   1  0.00    931.33  96 87.21

The copy job proceeded to completion without issue. This was all done over gigabit Ethernet from a Synology DS412+ NAS; note the SSD is able to ingest files at what amounts to line speed (gigabit Ethernet maxes out at about 120-ish MB per second, factoring in a bit of overhead), while the slower 5400-rpm HDD can’t write nearly as fast.

I was interested in seeing what FD would do after the job was completed, and the system didn’t disappoint. There was a pause of about twenty seconds, and then FD began moving data. Based on the observed throughput and the amount of data moved, it’s clear that FD was transferring things from the SSD to the HDD:

      disk0              disk1 
  KB/t tps  MB/s     KB/t tps  MB/s 
  4.00   1  0.00  1010.35  75 73.90 
  4.00   3  0.01  1024.00  72 71.94 
  4.00   1  0.00  1009.97  73 71.90 
  5.33   3  0.02   932.87  78 70.99 
  8.00  38  0.30  1011.09  79 77.91 
  4.80  10  0.05   969.86  56 52.98 
  0.00   0  0.00   999.47  75 73.14 
  4.00   1  0.00   404.18  88 34.70 
 30.72 304  9.11   748.71  68 49.66 
 30.29 295  8.71   484.47  93 43.94 
 29.43 318  9.13     0.00   0  0.00 
 29.89 316  9.21     0.00   0  0.00 
 30.78 289  8.68     0.00   0  0.00 
 20.92 646 13.20     0.00   0  0.00 
 29.62 468 13.55     0.00   0  0.00 
 23.18 428  9.70    75.83  46  3.40 
 29.27 406 11.62     0.00   0  0.00 
 29.69 239  6.92     4.00   1  0.00 
 28.89 369 10.40     0.00   0  0.00 
 29.09 517 14.68     0.00   0  0.00 
      disk0              disk1 
  KB/t tps  MB/s     KB/t tps  MB/s 
 30.12 455 13.39     0.00   0  0.00 
 28.90 587 16.57     0.00   0  0.00 
 30.40  25  0.74    69.92  50  3.41 
  0.00   0  0.00     0.00   0  0.00 
 13.33 156  2.03     6.67   3  0.02 
  0.00   0  0.00     0.00   0  0.00 
  8.00   9  0.07     0.00   0  0.00 
  0.00   0  0.00     0.00   0  0.00 
  0.00   0  0.00     0.00   0  0.00 
128.00 202 25.21   128.00 202 25.21 
128.00 455 56.90   128.00 454 56.78 
125.69 359 44.12   128.00 353 44.18 
111.18 427 46.39   128.00 367 45.92 
128.00 485 60.64   128.00 484 60.52 
119.74 667 78.00   128.00 622 77.76 
124.13 681 82.54   128.00 659 82.36 
128.00 665 83.13   128.00 665 83.13 
123.59 597 72.06   128.00 575 71.89 
128.00 679 84.88   128.00 678 84.75 
123.42 683 82.31   128.00 658 82.25 
      disk0              disk1 
  KB/t tps  MB/s     KB/t tps  MB/s 
123.56 676 81.56   128.00 650 81.24 
128.00 503 62.91   128.00 503 62.91 
122.32 506 60.47   128.00 483 60.41 
128.00 643 80.38   128.00 642 80.25 
122.73 682 81.74   128.00 652 81.50 
123.80 685 82.82   128.00 662 82.75 
128.00 565 70.64   128.00 564 70.52 
122.57 529 63.34   128.00 506 63.28 
128.00 555 69.40   128.00 555 69.40 
121.91 708 84.28   128.00 672 84.00 
128.00 661 82.63   128.00 661 82.63 
122.28 586 69.96   128.00 558 69.74 
128.00 458 57.28   128.00 457 57.16 
108.32 381 40.34   127.76 319 39.86

This was correlated with fs_usage, which showed reads from disk0 and writes to disk1, along with what are obviously Core Storage calls occurring between each:

15:33:27.189432      RdChunkCS     D=0x00debd00  B=0x20000      /dev/disk0s2
0.000328 W kernel_task.17601

15:33:27.189468    RdBgMigrCS      D=0x0033d8a0  B=0x20000      /dev/CS
0.000367 W kernel_task.17601

15:33:27.190136      WrChunkCS     D=0x0aeac500  B=0x20000      /dev/disk1s2
0.000629 W kernel_task.17601

15:33:27.190172    WrBgMigrCS      D=0x0033d8a0  B=0x20000      /dev/CS
0.000670 W kernel_task.17601

15:33:27.190531      RdChunkCS     D=0x00debe00  B=0x20000      /dev/disk0s2
0.000328 W kernel_task.17601

15:33:27.190568    RdBgMigrCS      D=0x0033d8c0  B=0x20000      /dev/CS
0.000369 W kernel_task.17601

15:33:27.191233      WrChunkCS     D=0x0aeac600  B=0x20000      /dev/disk1s2
0.000627 W kernel_task.17601

15:33:27.191269    WrBgMigrCS      D=0x0033d8c0  B=0x20000      /dev/CS
0.000667 W kernel_task.17601

15:33:27.191638      RdChunkCS     D=0x00debf00  B=0x20000      /dev/disk0s2
0.000336 W kernel_task.17601

15:33:27.191673    RdBgMigrCS      D=0x0033d8e0  B=0x20000      /dev/CS
0.000376 W kernel_task.17601

The four calls listed, “RdChunkCS,” WrChunkCS,” “RdBgMigrCS,” and “WrBgMigCS,” are all referenced in the fs_usage source as being Core Storage operations; the last two are noted in the code comments as referring to “composite disk block migration” calls. This is the actual “tiering” activity that forms the core of Fusion Drive. The blocks being referenced are sequential (on disk0, the process is reading block 0x00debd00, then 0x00debe00, and then 0x00debf00), and the actual chunk size being moved around is 128KB (0x20000 bytes).

The FD migration process ran for a short amount of time (though longer than the iostat output above), then stopped. I totaled up the amount of data copied, and it came out to approximately 4GB.

I repeatedly copied large files to the drive, and the results remained consistent—the copy operations would land on the SSD until it filled, then shift seamlessly over to the HDD and continue until completion. The SSD would then immediately move data off of itself until it had 4GB of free space. If I copied in a 2GB file, it would demote 2GB of data from somewhere else to the HDD; if I copied a 1GB file, it would demote 1GB. If I copied in 50GB, it would smoothly ingest all 50GB, and then demote 4GB of data off of the SSD.

This 4GB buffer is definitely what Anandtech’s source meant. FD doesn’t keep a constant 4GB “landing zone” if the amount of available SSD space is greater than 4GB; rather, FD keeps a minimum of 4GB free on the SSD.

But what goes where?

What about new files? Where are they created? Where are App Store apps installed? What about non-App Store apps? What about video capture? What about files downloaded in a web browser? Where does all of this stuff go?

The answer is simple: it all goes to the SSD, and there is nothing you can do to change that.

There are no options to configure, no pinning settings to adjust, and no user-visible method to decide what goes where. The FD volume is a single volume, and its Core Storage underpinnings direct all IO to the SSD first. New files are saved transparently to the SSD side of the Fusion Drive, as are new applications you install. Everything goes to the SSD first.

The logic behind this is clear: Fusion Drive is not meant to be a feature that appeals to the propeller-head geek. The kind of person who already has an SSD and a spinny disk in his Mac… and who symlinks his iTunes and iPhoto libraries off the HDD onto the SDD… and who enjoys meticulously balancing out which files go where will almost certainly not enjoy Fusion Drive’s hands-off approach. Fusion Drive is not designed to be poked at or prodded. Rather, much in the same way that Time Machine’s hands-off approach brought backup to people who otherwise wouldn’t be bothering, Fusion Drive’s hands-off approach brings tiering to Mac masses who otherwise can’t be bothered. The presentation is very Apple-like, with no knobs to twiddle.

Promotions

It is annoyingly difficult to actually track what gets promoted up to SSD and down to HDD with FD, because the tools to correlate exactly which 128KB chunks FD is moving to their parent files are difficult to use. However, Patrick Stein’s experiments with dd demonstrated that it is relatively simple to force FD to promote 1MB chunks up the SSD. After some quick poking and prodding, it’s similarly easy to force FD to move entire files up, as well.

It doesn’t appear to take much. I used cat to send a 600MB file to /dev/zero and ran purge in between each operation to ensure the disk cache was flushed. The file was definitely located on the HDD and not the SSD, as iostat showed all of the IO activity occurring on disk1. After doing this four times, the SSD lit up, with fs_usage showing lots of sequential Core Storage reads and writes. All 600MB were moved to the SSD, and the fifth time I ran cat file.blb > /dev/zero, iostat showed all of the IO occurring on the SSD instead.

That was the most number of times I had to try to promote something, too. Subsequent tests with different files (all large files, between 800MB and 2GB) all showed the whole-file tiering kicking off after two full reads instead of four. With every file except the initial one, the files were promoted up to SSD after just two reads and the third read was from SSD. At the same time, since the SSD was at capacity from my earlier testing, an equal amount of data was demoted down to HDD shortly after.

Unfortunately, careful monitoring of fs_usage didn’t really seem to show anywhere in the file system where Core Storage was keeping its usage data. I kept an eye on file system reads and writes as all of these promotions and demotions, but I didn’t catch any specific location that might be in use on the filesystem. It’s definitely not using Hot File Clustering (OS X’s equivalent to disk defragging); the .hotfiles.btree file is not present on the FD volume, which is consistent with past behavior on SSD-equipped Macs. The underlying mechanism still remains to be explored, and we’re keeping our eyes out for additional technical documentation on the hows and whys.

Camping out

Commenters in our previous articles had a lot of questions on how Boot Camp works with FD. The answer is that it works fine, with certain caveats as outlined in Apple’s tech document—you cannot use AFD with 3TB drives, at least not yet, and you must use the Boot Camp Assistant to create your Windows partition.

There are some additional restrictions on FD partitions, too. FD supports only one additional partition on the HDD and none on the SSD, so if you’re planning on using Boot Camp, the Boot Camp partition will be the only modification to the entire volume’s layout that you’re allowed to make.

Boot Camp Assistant works without any issue, creating a bootable USB thumb drive using a Windows 7 ISO. You can use Windows 8, though Apple hasn’t yet released Windows 8 drivers for all of its hardware, so some of the system’s devices might not work correctly or at all. I used a Windows 7 Ultimate ISO from my Technet subscription, which the Boot Camp assistant happily accepted.

Creating a bootable USB disk to install Windows 7.

Partitioning is interesting. As is standard with Boot Camp, you’re given a graphical slider choose the size of your Windows partition. However, the minimum size you can make your OS X partition is 129GB, which makes sense given that your Boot Camp partition must be created on the HDD and cannot be on the SSD.

You can’t have an OS X partition smaller than the SSD plus a small amount of HDD space.

After the partition is created, you’re rebooted into the Windows setup process, which boots off of the USB stick you created in the previous step. The process from there doesn’t differ from how Boot Camp has been for years, though the Windows setup wizard does give you the ability to cheerfully destroy your entire OS X setup with a misplaced couple of clicks. You’ll want to make sure to pick the right partition on which to install things. As is standard, Boot Camp names the partition “BOOTCAMP.”

Install Windows here. I had to format the volume NTFS first, which is par for the course with Boot Camp. Installing Windows elsewhere could lead to problems.

After this, Windows setup runs through all of its steps and reboots, and I was dropped directly onto the desktop with no complaints. Explorer shows only a single drive—as with OS X, there’s just one volume visible, but here in Windows it’s merely the Boot Camp partition on the HDD. The Disk Management MMC snap-in shows more details on the layout of the SSD and the HDD, though you won’t be putting any data on anything except your Boot Camp partition. Nothing else is accessible, and modifying anything here could lead to data loss on the OS X side of the house.

The MMC shows the SSD and the HDD, but you won’t be doing anything to them.

Breaking things

It’s clear a loss of either the HDD or the SSD would spell disaster for the aggregated Core Storage volume that makes up Fusion Drive. The Recovery Partition, standard on Macs since OS X 10.7, is located on the HDD. This means that if the SSD is ever lost, you’d still at least be able to boot and run some basic diagnostics at the command line. Apple’s own troubleshooting training has some more details, but most of the “repair” steps to fix a problematic Fusion Drive volume are destructive, requiring a rebuilding of the Core Storage volume.

I booted into the Recovery Partition (which you can do by holding Command-R at boot) with the intent of seeing if I could break the Core Storage volume into its components and then actually install OS X on one or the other. This required a bit of work at the command line, but separating the volumes was pretty easy:

Don’t do this unless you really, really mean to.

After this, the Mountain Lion installer saw both disks separately.

Both physical disks are visible separately after removing the Core Storage stuff. We have achieved…fission?

I didn’t proceed to installation, but it would have been a simple matter to do so at that point. Instead, I wanted to see what would happen if I rebooted; with the Recovery Partition living on the HDD but the boot partition nominally living on the SSD, I was curious about what would happen. Upon rebooting and holding down alt to see my choice of boot drives, I was greeted with a lonely Boot Camp partition.

A missing OS X partition is ordinarily a bad thing.

Attempting to boot back to the Recovery Partition with Command-R yielded the spinning globe of Internet Recovery, which took a few minutes to download. Once that had booted, I decided to follow Apple’s instructions on how to restore things in the event of trouble, because I had a totally non-working OS X installation. Apple’s directions in the earlier slides indicate that the version of Disk Utility shipping with 10.8.2 is able to re-create the Core Storage setup, and this functioned as advertised: I started Disk Utility and it immediately prompted me to repair my broken volume, with a warning that the repair process would be destructive:

Disk Utility can make it all better again.

Once this completed, I reinstalled Mountain Lion on the reconstituted volume and was very shortly back at an unsullied fresh OS X 10.8.2 desktop, with all my data gone.

The takeaway here is that users are clearly not meant to be fiddling around with the disk subsystem in Fusion Drive. It is possible to strip all of the Core Storage stuff, by deleting the logical volume and its container logical volume group, but doing so doesn’t appear to leave you with a bootable system. It is probably possible to install OS X on the SSD at that point and use the Mac normally, though I’m not entirely sure why you’d want to do so. There aren’t really any advantages to self-managing data placement on the SSD versus the HDD. Fusion does a good job of keeping things fast.

What happens next

Fusion Drive appears to work as advertised, though there are still unanswered questions. Observation and experimentation takes us only so far; I very much would like to get my hands on actual developer documentation explaining the parameters under which Fusion Drive operates. Particularly, I’d like to see what mechanism it’s using to determine which chunks are tiered down; it’s metadata-based, but on what metadata? Where is the tiering information kept? It moves 128KB chunks, but does that mean it’s capable of actually tiering individual 128KB chunks based on need? We know that it works with chunks at least as small as 1MB, but what is the lower limit?