Mac OS X 10.4 Tiger

Introduction

Back in early November of 2003, I introduced my Mac OS X 10.3 Panther review with some concerns about Apple's OS release cycle.

It's strange to have gone from years of uncertainty and vaporware to a steady annual supply of major new operating system releases from Apple. But do I really want to pay US$129 every year for the next version of Mac OS X? Worse, do I really want to deal with the inevitable upgrade hassles and 10.x.0 release bugs every single year? Is it worth it, or is a major OS upgrade every year simply too much, too often?

In the end, I concluded that I was okay with yearly releases, but that some sort of adjustment for "normal" customers would be nice.

If there's going to be any consumer backlash, it's not going to start with me. I think Panther is worth the cost, but I consider its price to be an investment in the future of Mac OS X—something I obviously have strong opinions about. I'm probably not a typical user, however. If Apple wants to help ease the burden of the larger Mac community, decent upgrade pricing would be a good start. With a yearly release schedule, that is nearly the same thing as a simple price reduction, but if so, so be it.

So convinced was I of the inevitability of the Mac OS X yearly release juggernaut that I never even considered the possibility that relief from the $129-per-year Mac OS X tax might come in the form of an extra six-month wait for version 10.4. "Let's do this again next year" were my exact words at the end of the Panther review.

Well, here we are 18 months and 6 days later, finally getting a look at Mac OS X 10.4 Tiger. Windows users patiently waiting for Longhorn may not be sympathetic, but the longer wait for Tiger is something new to Mac OS X users.

Tiger's longer gestation doesn't mean that the rate of change has slowed, however. Tiger includes updates that are at least twice as significant as any single past update. Mac OS X is now getting to the point where significant improvements require a larger time investment. As far as the core OS is concerned, most of the low-hanging fruit has been harvested. Now it's time for Apple to get down to the real work of improving Mac OS X.

Tiger also represents a milestone in Mac OS X's development process. Apple has promised developers that there will be "no API disruption for the foreseeable future." Starting with Tiger, Apple will add new APIs to Mac OS X, but will not change any existing APIs in an incompatible way. This has not been the case during the first four years of Mac OS X's development, and Mac developers have often had to scramble to keep their applications running after each new major release.

Despite its NeXTSTEP roots, Mac OS X is still a very young operating system. Most of the technologies that make it interesting and unique are actually brand new: Quartz, Core Audio, IOKit, Core Foundation. The hold-overs from NeXT and classic Mac OS have also evolved substantially: QuickTime, Carbon, Cocoa.

It's tempting to say that Tiger marks childhood's end for Mac OS X, but I think that goes too far. A more accurate analogy is that Mac OS X versions 10.0 through 10.3 represent "the fourth trimester" for Apple's new baby—a phrase used to describe the first three months of human life, during which the baby becomes accustomed to life outside the womb. As any new parent knows (yes, I am one of them), this is not an easy time of life, for the baby or for the parents.

It's been a rough journey, but we've made it through intact: Apple, Mac OS X, and Mac users everywhere. Tiger has arrived. Let's see what this baby can do.

Background

Ars Technica has been following Mac OS X since the second developer release in 1999. Earlier articles are listed below in reverse-chronological order, with the major public releases in bold.

• Mac OS X 10.3 Panther 11/09/2003
• Mac OS X 10.2 Jaguar 09/05/2002
• Mac OS X 10.1 10/15/2001
• Metadata, The Mac, and You 8/20/2001
• Mac OS X 10.0 4/02/2001
• MWSF: Mac OS X Post-Beta 1/17/2001
• Mac OS X Public Beta 10/03/2000
• Mac OS X Q & A 6/20/2000
• Mac OS X DP4 5/24/2000
• Mac OS X DP3: Trial by Water 2/28/2000
• Mac OS X Update: Quartz & Aqua 1/17/2000
• Mac OS X DP2: A Preview 12/14/1999

The earlier articles contain information that will not be repeated here, as well as concepts that will be referenced but not re-explained. In general, it is safe to assume that any praise or criticism found in the 10.3 article that is not either reinforced or contradicted in this article continues to be true of 10.4.

My impressions of Tiger are based on my experience using it with the following hardware:

• Dual G5 2GHz: A dual 2GHz Power Mac G5 with 2.5GB RAM, an ATI Radeon 9800 Pro video card with 128MB of VRAM in an 8x AGP slot, two 160GB 7,200 RPM Serial ATA hard drives, a DVD-RW/CD-RW drive, and a 23-inch Apple Cinema HD Display (the non-aluminum one).
• Dual G5 1.8GHz: A dual 1.8GHz Power Mac G5 with 1.25GB RAM, an ATI Radeon 9600 XT video card with 128MB of VRAM in an 8x AGP slot, a 80GB 7,200 RPM Serial ATA hard drive, a CD-RW/DVD-ROM drive, and a Sony E540 21-inch flat screen CRT.
• 15" AlBook: The 1GHz 15-inch Aluminum PowerBook G4 which I reviewed here at Ars Technica.

Long-time readers will note that my faithful blue and white G3/400, which has participated in all earlier reviews of the major releases of Mac OS X, has finally been retired. My "low-end" test system now sports a 1GHz G4 with 512MB of RAM, and the high-end system now has another hard drive and an extra 1GB of RAM. I think the upgrades are fitting, given the 18-month gap between Panther and Tiger. I only regret that I no longer have the absolute fastest Apple system to test with, since it adds that extra little bit of legitimacy to my performance gripes.

Branding

The promotion of the "big cat" names, Jaguar, Panther, and now Tiger, as official parts of the product naming scheme for Mac OS X releases leads me to wonder if Apple will eventually create a new set of actual, internal code names for its future OSes. If it happened for Tiger, I don't know about it. (On the other hand, there was "Smeagol" for the special G5-only build of 10.2.7.)

The semi-literal fur interpretation passed up by Panther's marketing team is also spurned by Tiger. Although a "tiger stripes wreathed in darkness" theme was used on pre-release DVDs and the associated banners at WWDC, the only tiger stripes you'll find in the official release are in a bundled desktop picture.

Instead, Tiger goes with a modified version of the "metal X" theme introduced with Panther. This time, the serifs are gone. In earlier variants, the ends of the "X" often extended past the edges of the display area. You can seen an example on the left below. The version on the right is the final version.

The other theme is the use of "spotlights," an obvious reference to the new search technology of the same name. There's even a new spotlight-themed introduction movie featuring our old friend the word "welcome" in its usual assortment of languages.

Finally, let's see how the update box art line-up looks.

Mac OS X boxes: Cheetah/Puma, Jaguar, Panther, Tiger

Well, the Tiger "X" does make the Panther "X" look kind of puny. But the spotlight effect lessens the "Box o' Death" thing Panther had going on. On the other hand, the Tiger section of Apple's web site is much less shy about using jet black all over the place. The Tiger box is not bad, but I'm kind of disappointed that there's no radical new design this time around. Maybe similar designs come in pairs?

Installation

Tiger comes on DVD-ROM media only. This is a first for Mac OS X. For customers without DVD-capable optical drives, Apple offers a $9.95 "Media Exchange Program." To take advantage of the program, customers must purchase Tiger, fill out a form, then mail it back to Apple along with their original Tiger DVD and a $9.95 payment (plus any applicable sales tax). That's pretty annoying, but I can understand Apple's decision to move to DVDs. Like the transition from ADB to USB that many Mac users considered abrupt and premature, I'm sure this will all be long forgotten by the time Mac OS X 10.5 arrives.

There's not much new to note about the installation and setup process. Like the later versions of Panther, the Tiger setup process includes a data migration tool that offers to import user accounts, network settings, and applications from another Mac connected via a FireWire cable.

I tried to use this feature once while setting up a new Panther machine, but the granularity of the choices made it useless to me. I wanted to migrate my account (preferences, settings, etc.) but didn't want to migrate any of my documents. The only choice that was close to what I wanted was the full home directory migration. At over 100GB, my home directory alone wouldn't have fit on the target machine.

Maybe my example isn't typical, but I'd still like to see an option to migrate, say, just my account and my "Library" directory. Unfortunately, Tiger doesn't include any new options in this area.

Tiger's new look

Apple and Spotlight
menu icons

Tiger doesn't match Panther in terms of the sheer number of seemingly arbitrary graphical changes, but it's got its share. To start, the Apple menu icon has changed yet again. It's now a brighter, flatter shade of blue. I think it's a bit gaudy, but it matches up nicely with the new icon in the opposite corner of the menu bar, a magnifying glass icon that we'll learn more about later. The graphite versions are a lot more subdued.

The biggest arbitrary graphical change in Tiger is the new unified titlebar/toolbar window style. It looks kind of like the titlebar simply grew downward, overwhelming the toolbar area.

The new unified titlebar/toolbar look

I have no idea what motivated this change, or why some applications use it and some don't. Mail uses it; TextEdit does not. System Preferences uses it; Disk Utility does not. Spotlight uses it; Preview does not. Xcode uses it, but not in every window. You tell me the pattern here.

If I had to guess what motivated the creation of this new style, I'd say it was done to eliminate a line (the line between the titlebar and the toolbar) that someone at Apple decided was unnecessary clutter, and to increase the size of the drag area for non-metal windows. These new windows can be dragged by clicking and dragging on any "gray" area in the titlebar or toolbar.

I'm not sure how smart it is to encourage users to try to drag by clicking in an "empty" area of the toolbar. The click activation regions for toolbar buttons are not always easy to predict. They often extend well beyond the bounds of the toolbar graphic itself. And even ignoring that, who wants to get in the habit of aiming between buttons anyway?

But who knows, maybe I will start dragging these windows using something other than the top 22 pixels that used to be the clearly delineated titlebar. After all, I got used to "sticky" pull-down menus too, right?

As for the elimination of that darned line, well, just look at how that worked out for menu separators. They disappeared in Mac OS X 10.0, much to my disappointment, only to make their triumphant return in 10.3. I am "pro line" in this case as well, although not strongly so.

Incidentally, this is not really a totally new window style. Instead, it's simply a variant of the normal (non-metal) Aqua window style. In Interface Builder, it shows up as a checkbox option called "Unified title/toolbar." The option applies to normal windows only; the checkbox is disabled for brushed-metal windows.

One application in particular deserves a special mention for its graphical sins: Mail, Apple's unimaginatively named e-mail client. It uses the unified titlebar/toolbar look, but that's not the problem. The problem is the buttons that sit in the toolbar. This is what the Mail toolbar buttons used to look like in Panther.

Panther Mail toolbar

Here's how they look in Tiger.

Tiger Mail toolbar

I don't know about you, but I like the Panther look a heck of a lot better. The first time I saw the new Mail toolbar, I filed a bug on it. (Radar 3968093: "Toolbar buttons in Mail 2.0 are hideously ugly.") It was immediately closed as a duplicate, so at least one other person agrees with me. (Maybe it was this person.)

The whole Mail application looks like it got beaten with the ugly stick in Tiger. Take a look.

Tiger Mail

(No offense to "the minds behind Tiger," by the way.)

Okay, granted, Apple did finally come to its senses and ditch the mailbox drawer, replacing it with a splitter pane on the left. But what's with the sudden attack of pale blues and grays?

Aesthetics are admittedly subjective, and I can imagine someone liking the new look. But the toolbar buttons suffer from some usability problems as well. First there's the issue of the little (pale blue, of course) "capsules" that surround each button. The capsules make all the toolbar buttons the same shape, and very similar in overall appearance. This flies in the face of Apple's own human interface guidelines, which state:

Each toolbar icon should be easily and quickly distinguishable from the other items in the toolbar. Toolbar icons emphasize their outline form, rather than subtler visual details. As shown in Figure 10-15, each Finder toolbar icon's shape is unique.

Doh!

There's also the strange case of the conjoined toolbar buttons. As the "customize toolbar" sheet shows, some buttons are available individually, some in groups, and some in both forms. This is inflexible, inconsistent, and again, a little strange.

Mail's "customize toolbar" sheet

I didn't mean to turn this section into a Mail 2.0 intervention, but honestly, it is the poster boy for unexpected graphical changes in the Tiger user interface. I'm all for experimentation; I just hope the next experiment turns out better than this one.

Kernel updates

The Mac OS X kernel has undergone some very significant changes in Tiger. The kernel is an interesting beast: a Mach microkernel fused to a traditional BSD-style system call interface, plus several entirely Apple-created technologies like IOKit. But for the purposes of this discussion, and from the perspective of most developers, the kernel can be seen as a single entity that just happens to have several different interfaces.

Interface rationalization

Unfortunately, historically, it's those interfaces that have been lacking in Mac OS X. There are many interesting kernel-related things that developers (including Apple) have needed to do to support their Mac OS X applications. In the early life of Mac OS X, it was all Apple could do to merely make these things possible. Later, efficiency was considered. But at every stage, developers were warned that many of the things they needed to do were not "officially supported" beyond the current major version of the operating system.

Apple did not discourage these "unofficial" activities, however. They recognized that it was their own failure to provide stable, official APIs for the required functionality that was the root cause of the current situation.

The message to developers was a bit of an odd one. "We at Apple recognize that you need to do this type of thing; we don't have a stable interface yet; you can use this API and dig into these data structures in the meantime; keep in mind that things may change in the next major revision of Mac OS X; if you can at all minimize how much you dig around in kernel data structures, please do so." The final bit of advice was always this. "Please tell us why you need to do what you're doing, and how we can make your task easier."

This went on for years, largely unknown to the casual user. Occasionally, users would have to download a new printer driver or VPN kernel extension when upgrading to a new major version of Mac OS X, but that's about it.

More significantly, the lack of official, stable kernel APIs also prevented some parts of the kernel from being fully explored by third-party developers. Take, for example, file system plug-ins, which were a much-vaunted feature of the Mac OS X kernel. Perhaps only the geekiest readers will recall the tantalizing prospect of layered, pluggable file system modules that was dangled before the Mac community in the early, pre-10.0 days of Mac OS X development. New file system support would be a dynamically-loaded module away! Encryption layers could be added with ease to any existing file system module!

It all sounded very promising, but the only vendor to really take advantage of these features was Apple itself. Few third-party developers were willing to commit to creating (and then supporting) something as mission-critical as a file system plug-in on top of unstable, "unsupported" kernel APIs and data structures.

Something else happened along the way. Despite its dire warnings to developers, Apple tried mightily to provide binary compatibility for kernel extensions (kexts) across major revisions of Mac OS X. The effort was largely successful, but at a tremendous cost.

In order to maintain binary compatibility, Apple had to forego many beneficial kernel changes. "Obvious" performance improvements had to be left unimplemented. Apple's ability to add new features was very limited. All changes had to be made cautiously, since Apple never knew which existing third-party kernel extensions were digging around in which kernel data structures. Worse still, some bugs actually could not be fixed because doing so would break binary compatibility with one or more important kexts. And even with all this effort, sometimes third-party kernel extensions broke anyway after major OS updates.

Obviously, this situation was not ideal. Parts of the kernel remained untapped, bugs were left unfixed, kernel improvements had to be postponed, and kexts still sometimes broke. But in the meantime, developers learned how to get most of their really necessary kernel-related things done in Mac OS X, and Apple learned what developers needed most, and how best to provide it.

With Tiger, Apple is finally ready to put some kernel interface stakes in the ground. For the first time, there are stable, officially supported kernel programming interfaces (KPIs). Even better, there's an interface versioning mechanism and migration policy in place that will ensure that the pre-Tiger situation never happens again.

From Tiger forward, kernel extensions will link against KPIs, rather than directly against the kernel. The KPIs have been broken down into smaller modules, so kexts can link against only the interfaces that they actually need to use.

Each KPI has a well-defined life cycle made up of the following stages.

• Supported - The KPI is source and binary compatible from release to release.
• Deprecated - The interface may be removed in the following major release. Compiler warnings are generated on use.
• Obsolete - It's no longer possible to build new kernel extensions using this KPI, but binary compatibility for existing kexts that use this KPI is still assured.
• Unsupported - Kexts using this KPI will no longer work, period.

The most significant part of this new system is that the kernel itself can and will change behind the scenes. KPIs will descend towards the "unsupported" end of the life cycle only as kernel changes absolutely demand.

Best of all, multiple versions of a KPI can coexist on the same system. This allows a KPI to move forward with new abilities and a changed interface without breaking kernel extensions that link to the older version of the KPI. The expectation is that the kernel can undergo a heck of a lot of changes while still supporting all of the KPIs.

The proof of that pudding is in Tiger itself. Though the Tiger kernel has radically changed behind the scenes, there are KPIs to support nearly all of the pre-Tiger "bad behavior" of kernel extensions. Yes, some kexts will break in Tiger, but that's nothing new for a major OS X release. This time, however, the future looks a lot brighter.

(Incidentally, the ability to dig right down into kernel data structures or link directly against the entire kernel still exists in Tiger. But it should rarely be necessary and is highly discouraged. Apple also says flat-out that any kext that does so is tying itself to a specific kernel version—not just a major version either, the exact version. Even so, linking against a specific kernel is actually reasonable in some rare situations, such as in-house, proprietary hardware development, so it's nice that it's still possible.)

There are a few more niceties in the world of kernel extensions that, surprisingly, are aimed at making them more user (yes, user, not developer) friendly.

Kernel extensions are now expected to include human-readable (and perhaps localized) product names. This information will be extracted for use in dialog boxes and other status items in the GUI. For developers that need several kernel extensions to implement their product, Tiger also supports the kernel extension grouping, so that the multiple kexts can be presented as a single product to the user.

Okay, I guess this is also a "developer friendly" change. Support calls are a lot easier to handle when the user can read "AcmeSoft Driver version 1.03" right off their screen and the developer can know exactly which set of kernel extensions they're talking about.

Overall, it's clear that with Tiger, Christmas has come early for kernel extension developers and users alike.

Concurrency

In earlier versions of Mac OS X, the kernel was essentially single-threaded. That is, only one thread could be in the kernel at any given time. Other threads could continue to run, so long as they did not run code that needed to execute inside the kernel.

For example, a fractal-generating application does not need to be in the kernel in order to do the calculations necessary to draw the next iteration of the fractal image. On the other hand, an application that wants to do any network, file or device i/o has to go through the kernel to do so.

The Mac OS X kernel uses locking to enforce these constraints. In Panther and earlier, any code that needs to call into the kernel to get its work done first has to grab the "kernel lock." If another thread already has the lock, the thread that wants it simply has to wait until it becomes available again. Now imagine 50 different threads all trying to get the kernel lock at the same time and you have what is known as "lock contention." The lock itself is also known as a "funnel" because it conceptually funnels all of the contending threads into a single, serialized stream of activity.

I described the pre-Tiger kernel as "essentially" single-threaded because it actually implements a "split funnel" system. As described earlier, a thread that wants to execute in the kernel needs to grab the kernel funnel. But if it needs to do network i/o only, it can give up the kernel funnel and pick up the network funnel. In other words, there's effectively one funnel for networking and one funnel for "everything else" in the kernel. As it turns out, that "everything else" is usually file i/o, so the kernel funnel is sometimes also called the file system funnel.

Even with the split funnel, this system is what is known as coarse-grained locking. The bigger the chunk of functionality protected by a lock, the coarser the lock is considered. And the coarser the locking, the more chance there is for contention. One "kernel funnel" and one "networking funnel" make for a very coarse setup indeed.

Matters get worse when you consider a multi-CPU system. More simultaneous threads vying for the same number of kernel locks means more contention. There are no Macs with more than two CPUs right now. But let's face it, a dual-core, dual-chip Mac is inevitable at this point. ("Confirmed!!!") Such a beast won't take too kindly to a split funnel kernel. What's an OS to do?

Finer-grained locking is the obvious solution. Instead of restricting access to huge chunks of the kernel, locks can be placed in front of smaller pieces of functionality. For example, instead of allowing only one thread at a time to do "any network i/o" in the kernel, smaller locks can be placed on the various parts of the networking stack: memory buffers, sockets, the protocol layer, etc.

Since there can be as many threads running in the kernel as there are individual locked pieces, more locks means more threads and a lower chance that a thread will want exactly the same lock as another thread. The end result is less contention, and the ability to scale to a higher number of CPUs (or "cores" with multi-core CPUs, or "threads" with symmetric multithreading).

On the other side of the coin, there is a measurable amount of overhead required to do the bookkeeping and synchronization for each lock. More locks means more overhead. At a certain point, more time is spent doing lock management and synchronization within the kernel than was saved by introducing finer-grained locking in the first place. So while extremely coarse locking like Panther's split funnel is obviously suboptimal, it's also unwise to go too far in the other direction, with tons of tiny locks on every single bit of kernel functionality.

The easiest way to determine the optimal scope for a kernel lock is to start coarse-grained (which is the easiest to implement) and then get progressively finer-grained until performance stops increasing under a typical load for that subsystem. This is the technique Apple used to develop the Tiger kernel. Tiger does away with the split funnel and replaces it with locks on the logical pieces of each kernel subsystem.

For example, a thread making a file i/o system call might start by grabbing the generic system call handler lock, then drop it and grab the lock on the file descriptor handler, then drop it and grab the lock on the kernel's file data structure access code, then drop it and grab the lock on the file system access code, and so on.

As soon as a thread drops a lock, another thread can pick it up, leading to a series of threads traversing well-defined pathways through each kernel subsystem, picking up and dropping locks as they go. In practice, it behaves a lot like "pipelining" does in a CPU, but has its own, different trade-offs for "pipeline length" (number of locks) and overall throughput.

As a consequence of the kernel's new multi-threading abilities, kernel extension writers have to be absolutely sure not to assume that they are the only process allowed in their part of the kernel at any given time. It's very easy to get lazy about this when the kernel is split into two giant pieces. Finer-grained locking in the Tiger kernel means more synchronization work for both Apple and third-party kernel extension developers. The benefits are well worth the effort, but expect some growing pains.

64-bit support

Panther introduced rudimentary 64-bit support to Mac OS X. It expanded the virtual address space (in the kernel, anyway) to 64 bits and allowed the use of 64-bit registers and the instructions that manipulate them (i.e., 64-bit math). But processes other than the kernel still saw a 32-bit address space. A single process could work with more than 4GB of memory (remember, the Power Mac G5 can hold up to 8GB RAM), but doing so required the programmer to manually juggle several 32-bit-addressable chunks of memory at once.

Tiger takes Mac OS X another small step in the 64-bit direction by allowing any process to see a 64-bit address space. Such a process must use 64-bit pointers in its code, of course, and that means that any libraries it uses must also be compiled to use 64-bit pointers.

In Tiger, the only 64-bit library is "libSystem," which is basically the BSD layer. A 32-bit version of libSystem is also included, of course (otherwise 32-bit applications would not run on Tiger).

A process can do a lot using only libSystem: file and network i/o, math, inter-process communication, er...more math. But the notable thing it can't do is any sort of GUI operation. (Curses doesn't count, sorry.)

For GUI applications that need to address more than 4GB of memory as part of their work, the recommended strategy in Tiger is to spin off a 64-bit "worker thread" that only uses libSystem and communicates its results back to the host application using one of the various inter-process communication mechanisms.

This confinement of 64-bit processes to a non-GUI jail might seem severe, but in practice, a large proportion of processes that need to be 64-bit are already "faceless" worker processes doing computations on behalf of some larger system: offline rendering, server processes, scientific computing, etc. It's also a step up from Panther's 64-bit support, which is something at least.

It's clear that the road to "full 64-bit support" will be a long one. There are few benefits to being a 64-bit process for the vast majority of GUI applications. Nevertheless, it's safe to assume that, eventually, all Macs will include 64-bit CPUs. The introduction of 64-bit versions of all Mac OS X subsystems (Carbon, Cocoa, Core Foundation, QuickTime, Quartz, etc.) seems inevitable.

I just wonder how much benefit there will be from introducing any of that support piecemeal. For example, it'd seem unfair for Carbon to be 64-bit while Cocoa was not. All the higher-level GUI libraries rely on lower-level services like Quartz and Core Foundation anyway. So it seems to me that the best move in the future will be to roll out a complete 64-bit system all in one shot. That's a tall order, which is why I think it'll be a while.

Kernel summary

With sexy dual-core CPUs on the way, the removal of the split funnel may seem like the most important kernel change in Tiger. But the introduction of stable kernel programming interfaces is the real hero here. Without KPIs, none of the radical kernel changes in Tiger would have been possible. The KPIs finally set Apple's kernel development team free to do all the things they've always wanted to do.

Although 64-bit support is usually listed as a kernel feature, the Tiger kernel has gone about as far as it can go in that area. The ball is in the user-space libraries' court now.

Out to launch

Preface

The topic of this section may seem esoteric and not worthy of extensive coverage. I chose to write about it not so much because it is such a significant part of Tiger, but because I think it highlights some important aspects of Apple's development process.

It's going to get very Unix-y, very fast. I'll explain as much as I can, but this won't be a full-fledged Unix indoctrination starting from first principles. If this all sounds like greek to you, you can always skip to the next section.

Background

In Unix in general, and in Mac OS X in particular, there are a lot of ways to launch programs in response to external events, or on a timed schedule. Common events include system startup, shutdown, and restart, user login, and incoming or outgoing network connections.

In Unix, startup, shutdown, and restart events are handled by a series of interconnected shell scripts stored under /etc/rc.d/. The exact path and contents of that directory tree vary. Some Unix flavors use a different mechanism all together, but most bear the signs of their BSD or System V ancestry.

Timed execution is usually handled by the cron or at daemons. Login time events are triggered by shell-specific dot-files stored in the user's home directory. Network connections may be handled by individual daemons, or by one of many possible "super daemons" (e.g., inetd) that multiplex network connections, triggering other daemons as appropriate. Mac OS X does all of the above, plus it has its own XML-driven mechanisms for system startup and user login items that are independent of the Unix layer.

All of these systems suffer from limitations. The network super daemons in particular are limited by assumptions about their job. Most super daemons are IP-centric and assume one network connection handle per daemon. They're usually system-wide, making them unsuitable to, say, launch a daemon on behalf of a particular user. They also tend to pass the buck when it comes to the realities of modern desktop computing. A PowerBook may go to sleep in one location and then wake up somewhere else on an entirely different network.

As new program-launching needs have arisen in the Unix world, the reaction has always been to modify or piggy-back on one of the existing systems. For example, the ssh-agent daemon (think "keychain for ssh passwords") needs to launch once, and only once for each user login session. Shell startup files are read every time a shell is started. Multiple terminal windows means multiple shells, but still only one login session.

The Unix community's solution was to make runtime checks at shell startup time to see if the ssh-agent was already running, and launch it if it was not. This passed from clever hack into common idiom as ssh-agent became more popular. This is a good example of "The Unix Way."

Whenever Unix vendors have bucked the trend and tried to build something more than just incrementally better than the existing mechanisms, they've met with instinctive resistance. Just look at one Unix user's assessment of the new service management system in Solaris 10.

My concern about SMF [Service Management Framework] is that its not as transparent as a system like rc.d on NetBSD/FreeBSD or even the old sysV init. It is still fairly easy to understand, but there is a level of "magic" that wasn't there before.

As a heavy Unix user myself, I can see where this person is coming from. But honestly, only someone absolutely immersed in Unix culture could ever consider the rc.d or System V init systems "transparent." SMF uses shell scripts, but also XML. It's a new twist to old-school Unix hackers, and the initial reaction is predictably cautious.

Apple went down the same path as Solaris 10, adding its own XML-driven mechanisms for startup and login. The Unix community mostly ignored these efforts. They had their shell scripts, super daemons, and elaborate /etc/rc.d/ scaffolding.

But Apple is not your typical Unix vendor. While the new XML-based systems were nice, they represented two more systems in an already crowded field. In Tiger, Apple decided to tackle the program launching problem again, this time wiping the slate clean.

Launchd

For Tiger, Apple created launchd: one launch daemon to rule them all. Launchd does the job of all of the existing program launching mechanisms, and does it in a way that puts the least possible burden on the programs that it launches. Processes spawned by launchd don't have to worry about "daemonizing" themselves, checking for dependencies, or relaunching or keeping communication handles alive in the case of a crash.

Launchd can launch programs in response to any of the events listed earlier, and it can do so on behalf of the system or an individual user. It will discover dependencies on its own and launch programs in parallel when possible. This is essential for fast system startup. Mac OS X's older startup items system did the same thing, but it had to be explicitly told the dependencies.

Launchd supports a messaging protocol to answer questions like, "How many users are connected to this daemon?" and "Have you shut down yet?" Program shutdown is another example of an area where "The Unix Way" is usually deemed "good enough" despite obvious deficiencies. Traditionally, Unix services are shut down by sending a signal to the process, waiting a little while, and then sending a more harsh signal just in case the service refused to shut down. This is barbaric, but necessary because there's no standardized messaging system for Unix daemons. Launchd recognized the need and filled it.

Apple has developed launchd as an open source project that it hopes will be adopted by the wider Unix community. To the average Unix hacker, launchd probably looks like a reinvention of the wheel. I think it addresses a problem the Unix community doesn't even know that it has. In this way it's much like Mac OS X itself. There was "Unix on the desktop," and then there was Mac OS X. You'd think that alone would have been a big enough wake-up call.

If I were working on a Unix-based operating system, I'd be borrowing ideas and code from Apple like there's no tomorrow. Apple has certainly been smart enough to pull in the opposite direction, basing a huge part of its OS (particular its server OS) on open source Unix projects. Apple has returned the favor by contributing to many of those projects: FreeBSD, gcc, KHTML/KJS, etc. When Apple comes calling with their own open source Unix creations, I think it's foolish not to pay attention.

Anyway, enough preaching. What launchd means to Mac OS X is that all the preexisting program launch facilities will slowly be migrated to launchd. This won't happen overnight, or perhaps not even in the next few years, but the groundwork has been laid. There are also plans to extend launchd to handle device events (e.g., plugging in a camera) and to further standardize not just the protocol but the contents of service messages.

Like lookupd before it, launchd tackles a long-standing Unix problem head on. It's unashamedly ambitious, doing exactly what Apple wants it to do without undue regard for how it has been done in the past. Even the names "lookupd" and "launchd" reflect the straight-forward, can-do attitude that Apple brings to all of its Unix-related development. "Good enough" is never good enough for Apple's core OS team, and I'm glad.

Metadata revisited

It's been a while since I've written about metadata in Mac OS X. First, a few brief definitions for newcomers to the topic.

• Colloquially, metadata is "data about data."
• File metadata is information associated with a file that is not logically part of the file's actual content (i.e., its data), although it may be physically.
• File system metadata is file metadata that is stored as part of the file system data structures dedicated to each file. A file's modification date is a common example of file system metadata, but the Mac has historically supported much more: type and creator codes, Finder flags, icon position, label, etc.

All of these terms are subtly different and are generally not interchangeable. I will try to use them very precisely in this article.

So enamored am I with the world of file metadata that I wrote an entire article about it back in the summer of 2001. In the early days of Mac OS X's development, it was difficult to divine Apple's intentions in this area. The file metadata mechanisms in the developer versions and early public releases of Mac OS X seemed to be more a product of their mixed Apple/NeXT heritage than a manifestation of a coherent plan.

But even several months before the Public Beta release, I was already growing somewhat concerned about the apparent intentions of the mothership. Reviewing what I wrote in June of 2000, my trepidation is clear. At that time, the only official Apple statement on the topic was that they "strongly encourage developers to use file extensions as alternative means for identifying document types."

The arrival of an actual 10.0 release on March 24th of 2001 was such a significant event that the metadata issue was swept away by larger concerns. "Will this transition work?" "Can I run my old apps?" "What the heck is the deal with the performance?"

But by the time version 10.1 was released in October of 2001, Apple's direction was clear. The decree was simple: all files created by Mac OS X applications must include a file name extension. (Cue dramatic music.) My worst fears were realized, and my reaction was swift and merciless.

I didn't stop there. I went on to formally propose an alternate course of action for Apple. I created an online petition (stop laughing) in support of the proposal that gathered almost 10,000 signatures and comments from a broad cross-section of the Mac community. Here's my favorite.

I'm confident that the folks at Apple will eventually recognize the role that basic metadata, such as type/creator, plays in the Macintosh user experience. When I was working on the Finder in 1982 and designed the type/creator, desktop database, and resource mechanisms, I wanted to make sure that the user would almost never have to care about what the file's type was, how it should be named, or what application was appropriate to edit it. At the same time, I felt that the Finder should allow users to override the Finder's application preference as needed; thus, drag-and-drop to an application was provided. In the following years the software designers at Apple have done beautiful work in maintaining this ease of use with the Finder and File Exchange. Promoting a least-common-denominator solution is inappropriate for a company sustained by innovation; Apple needs to embrace the entire concept of metadata for user information, and consider type/creator as the beginning of something much more powerful for OSX.

Bruce Horn
Member of the original Macintosh development team.
Signature 252.

I filed the metadata proposal as an official bug (2826368) in Apple's bug tracking system and waited for what I hoped would be some sort of change in direction from Apple—or at least an acknowledgment of the issue.

In early 2002, Apple hired Dominic Giampaolo, the creator of the Be File System. BFS was one of the most advanced personal computer file systems of its time, with support for arbitrarily extensible file system metadata and efficient, real-time search queries. Dominic wrote an excellent book about BFS, and file system design in general. The book is out of print now, but it's available as a free download. Even today, BFS's features and performance are not matched by many commonly used file systems.

Dominic's arrival at Apple was seen as a sign of things to come. It doesn't take much to get Mac enthusiasts excited about future possibilities. Fair or not, expectations were high.

In August of 2002, Mac OS X 10.2 Jaguar was released with no significant changes to the file metadata policies or mechanisms. Two days after the release of Jaguar, Apple closed my metadata proposal bug in its bug tracking system, assigning it the status message "Closed/Behaves correctly."

This put quite a damper on the whole issue of Mac OS X metadata. The irrational Giampaolo mania of the Mac metadata prognosticators waned considerably. Having said everything I had to say on the topic, and having done everything I thought I could to effect some sort of change inside Apple, I largely abandoned the issue as well. What more could be done? Objections had been voiced, advice given, and file system gurus hired. All we could do was wait.

Mac OS X 10.3 Panther arrived in October of 2003. It contained no significant changes to the file metadata policies or mechanisms. (Noticing a pattern here?) So we waited some more.

The year 2004 arrived without the expectation of a new major version of Mac OS X until at least early 2005. Anyone who still managed to maintain an interest in the topic had to face a grim reality: Despite all efforts and expectations, the file metadata situation in Mac OS X had not significantly changed since version 10.1 in 2001.

A new hope

In March of 2004, Apple quietly released an update to its HFS Plus Volume Format technote (TN1150). The update described a new variant of HFS+ called HFSX.

HFSX is an extension to HFS Plus to allow additional features that are incompatible with HFS Plus. The only such feature currently defined is case-sensitive filenames.

Case-sensitive HFS+ (as it is called in end-user interfaces like Disk Utility) was actually introduced in Mac OS X 10.3 Panther, but was only exposed in the GUI in Mac OS X Server. Still, the new name and the stated intent to "allow additional features that are incompatible with HFS Plus" showed that Apple was willing to do something in the realm of file system technology.

Then there was the section named "Future Support for Named Forks" which contained these refreshingly non-archaic notions about file system metadata.

A number of products have implemented special-purpose solutions for storing their file- and directory-related data. But because these are not managed by the file system, they can become inconsistent with the file and directory structure.

HFS Plus has an attribute file, another B-tree, that can be used to store additional information for a file or directory. Since it is part of the volume format, this information can be kept with the file or directory as it is moved or renamed, and can be deleted when the file or directory is deleted.

Then things started to really heat up.

The contents of the attribute file's records have not been fully defined yet, but the goal is to provide an arbitrary number of forks, identified by Unicode names, for any file or directory.

(Emphasis added.) Now granted, it's been widely known for many years that HFS+ could technically support an arbitrary number of named forks, but the language in the technote has slowly shifted in its views towards these capabilities. After the HFSX update, the intent seemed very clear.

Still, alternate data streams ("forks" in Mac parlance) have always seemed like a very heavyweight implementation of arbitrarily extensible metadata. This is another reason why HFS+'s ability to support multiple named forks hadn't been much more than a curiosity in the past. Although the technote language changes renewed some interest, the technical issues seemed to remain.

But there was one final tidbit of information in the technote. Although it states that "the contents of the attribute file's records have not been fully defined yet," there was enough documentation to get an idea of what was to come.

Each [attributes file] record starts with a recordType field, which describes the type of attribute data record. The recordType field contains one of the following values.

enum {
    kHFSPlusAttrInlineData  = 0x10,
    kHFSPlusAttrForkData    = 0x20,
    kHFSPlusAttrExtents     = 0x30
};

The values have the following meaning:

kHFSPlusAttrInlineData

Reserved for future use.

kHFSPlusAttrForkData

This record is a fork data attribute. You can use the HFSPlusAttrForkData type to interpret the data.

kHFSPlusAttrExtents

This record is an extension attribute. You can use the HFSPlusAttrExtents type to interpret the data. A record of type kHFSPlusAttrExtents is really just overflow extents for a corresponding record of type kHFSPlusAttrForkData.

(Again, emphasis added.) Inline attributes. Now we were getting somewhere. While entire forks seemed like overkill for small pieces of data, the word "inline" sounded suitably lightweight. The phrase "reserved for future use" increased the metadata expectations for Tiger.

As it turns out, those expectations would be met in some very interesting ways. Starting in mid-2004, Apple began to publicly demonstrate and market some significant new metadata-related features scheduled for Tiger. If you're a casual Mac user, or maybe not even a Mac user at all, you may think you already know where this is going. Think again.

Extended attributes

Tiger includes support for the "xattr" family of functions in the BSD layer: getxattr(), setxattr(), removexattr(), listxattr(), plus "f"-prefixed variants that deal with file descriptors instead of paths. The "xattr" functions have existed in various forms for many years on other flavors of Unix, but this is their first appearance in Mac OS X. The Unix manual page explains.

Extended attributes extend the basic attributes associated with files and directories in the file system. They are stored as name:data pairs associated with file system objects (files, directories, symlinks, etc).

An extended attribute's name is a simple NULL-terminated UTF-8 string. Value is a pointer to a data buffer containing textual or binary data to be associated with the extended attribute.

In case it hasn't clicked yet: after over four years of waiting, Mac OS X now includes support for arbitrarily extensible file system metadata.

(I'm a big enough geek that I wanted to end that sentence with an exclamation point in addition to making it bold. I've decided to spare the casual reader. But if you read it that way on your own, right on, brother.)

Yes, it's really true. I commissioned long-time Macintoshian Achaia regular Marquis Logan ("nibs") of The Big Nerd Ranch to create a simple command-line tool that lets you see for yourself. He went beyond the call of duty, wrapping the BSD APIs in a set of CoreFoundation-like classes. You can download the source or the binary. (You'll need Xcode to build the source.) Here's an example of the program, appropriately named xattr, in action.

First, create a small text file to play with.

% cat > file
Testing 1 2 3.
^D
% cat file
Testing 1 2 3.

Next, add, list, modify, and delete some arbitrary attributes.

% xattr --set name John file
% xattr --set color red file
% xattr --list file
file
color   red
name   John
% xattr --set color blue file
% xattr --list file
file
color   blue
name   John
% xattr --delete color file
% xattr --list file
file
name    John

There's nothing up my sleeve, honest.

% ls -la
total 8
drwxrwx---    3 john  john   102 Mar 24 21:36 ./
drwxr-xr-x   66 john  john  2244 Mar 24 21:36 ../
-rw-rw----    1 john  john    15 Mar 24 21:36 file

No, there's no resource fork either.

% ls -l file/..namedfork/rsrc
-rw-rw----   1 john  john  0 Mar 24 21:36 file/..namedfork/rsrc

In Tiger, most Unix command-line tools are metadata-savvy too...

% cp file file2
% xattr --list file2
file2
name    John

...but some require an extra flag. (RTFM to be sure.)

% mkdir backup
% cp file backup
% rsync -E file backup
% xattr --list backup/file
backup/file
name   John

This is all happening on an HFS+ volume—not "HFSX" but plain old HFS+. As foreshadowed in March of 2004, an unmodified HFS+ volume does indeed have the ability to store an arbitrary number of attributes. The xattr APIs in Tiger (finally) leverage this ability. No new volume format has been introduced, and no reformatting is required.

Support on lesser volume formats is provided by the venerable dot-underscore file.

% hdiutil create -size 50K -fs MS-DOS -volname DOS dos.dmg
created: /Users/john/dos.dmg
% hdiutil attach dos.dmg
/dev/disk5                      /Volumes/DOS
% cd /Volumes/DOS
% touch file
% ls -l
total 0
-rwxrwxrwx   1 john  john  0 Mar 24 22:07 file
% xattr --set name John file
% xattr --list file
file
name    John
% ls -la
total 49
...
-rwxrwxrwx   1 john  john    4096 Mar 24 22:07 ._file
-rwxrwxrwx   1 john  john       0 Mar 24 22:07 file

As in earlier versions of Mac OS X, files prefixed with "._" contain all of the metadata for their un-prefixed partner that the native volume format cannot contain in a single file: resource forks, flags, dates, and now in Tiger, extended attributes.

The extended attribute APIs hide these details, and the other command-line tools have been updated to use the new APIs when needed. For example, look at what happens when the cp command is used to copy a file with extended attributes ("file2" from the earlier example) from an HFS+ volume onto a FAT (MS-DOS) volume.

% cd /Volumes/DOS
% cp /Users/john/file2 .
% xattr --list file2
file2
name    John
% ls -la
...
-rwxrwxrwx   1 john  john    4096 Mar 24 22:09 ._file2
-rwxrwxrwx   1 john  john      15 Mar 24 22:09 file2

Boom. Instant dot-underscore file. (Now there's a demo you'll likely never see Jobs do in a keynote presentation.)

Extended attributes: uses and limits

The Tiger implementation of extended attributes is "preliminary" in many senses of the word. There are some limitations on the number and size of attribute names and values. Attribute names can be up to 128 bytes long and are made up of Unicode (UTF-8) characters. Values are up to 4KB of arbitrary data. (There is actually no hard limit on the size of attribute values, but Apple recommends staying below 4KB for now.)

While there appears to be no hard limit on the number of attributes that can be associated with a file, the time it takes to add an attribute increases tremendously once a few thousand attributes are added. Running something like this

% touch file
% perl -e 'for(1..50000) { system "xattr --set a$i v$i file"; $i++ }'

for a few minutes results in a file with about 7,000 attributes. I thought maybe the attribute insertion performance was falling off due to hashing collisions in the implementation, so I tried adding a random string to the front of each attribute name. The performance didn't really improve. I'm not sure what the problem is. I ran it some more and got a file with over 12,000 attributes, but it took a while.

Interestingly, only the addition of attributes seems to slow down so precipitously. Listing all 12,000+ attribute names and values was nearly instant. Deleting an attribute was similarly quick.

This kind of torture testing may seem academic, but I'm trying to get a feel for how well HFS+ fares in its new role. I had hoped for, requested, and even now (in optimistic moments) still expect a new high-performance metadata-enabled file system from Apple. Tapping HFS+ to do the job in the meantime is fine with me, but I can't help pining for some more of the strengths of the Be file system to make their way to the Mac platform. (Of particular note is BFS's support for the creation of file-system-resident indexes on attributes, and its real-time search and notification features. But I'll have more to say about those topics later.)

The 128-byte limit on attribute names is reasonable, but remember that there is just one flat namespace for attributes. Some sort of policy for isolation is necessary; we can't have 5 programs all trying to write to the "name" attribute, after all.

To solve this problem, Apple recommends the use of a reverse-DNS naming scheme (e.g., com.apple.itunes.artist), but with the dots replaced with underscores in order to avoid conflicting with the namespace convention used by Cocoa's key/value coding system.

The final, and perhaps most important limitation of Tiger's extended attribute implementation can also be seen as a feature. The extended attribute APIs exist only at the BSD layer. There are no Carbon, Cocoa, or even Core Foundation interfaces to them at this time. If an application wants to use extended attributes, it has to use the (decidedly low-level) BSD APIs.

Historically, Apple has introduced many technologies "from the bottom up," providing BSD implementations first and adding the higher-level APIs later. The BSD layer usually has to come first, or at least simultaneously, since most high-level APIs in Mac OS X eventually trigger one or more BSD-level function calls. This is especially true for file i/o.

In Tiger, extended attributes are essentially being "field tested." The APIs exist, HFS+ supports them natively, and most BSD-level tools have been updated to honor them. But until there are Carbon, Cocoa, or at least Core Foundation APIs that abstract and explicitly leverage these new features, extended attributes will likely fly under the radar of traditional (GUI) Mac OS X applications.

That's a shame because there are a ton of potential uses for extended attributes, as the next topic will demonstrate.

Access control lists

Access control lists, or ACLs, are a finer-grained, more flexible way to control file permissions: who can do what to which files. In Tiger, ACLs are a supplement to the traditional Unix file permissions. Since I've never covered Unix file permissions in a Mac OS X article before, I'd like to do so now. Feel free to skip this section if you already know this stuff.

Unix file permissions

In Unix, each file belongs to a single user and a single group. Unix file permissions control the ability to perform three actions: read from a file, write to a file, and execute a file. For a given file, the permissions allow or deny each of these three actions for the user who owns the file, the group the file belongs to, and everyone else. That's a total of nine permissions: read, write, and execute for owner, group, and other.

In the text-based interface of Unix, the actions are traditionally abbreviated by the letters "r", "w", and "x." The file's owner and group are also displayed. Here's an example:

The file above belongs to the user "john" and the group "staff." The first character of the permissions string indicates the file type. It is "-" for normal files (as shown above), "d" for directories, "l" for symbolic links, and so on. The example above grants read, write, and execute permissions to the file's owner (john), group (staff), and "everyone else." Here's a more restrictive example:

In this example, the owner (john) can read and write to the file, the group staff can only read it, and everyone else cannot read, write, or execute it. The "-" characters within the permissions indicate a lack of a particular right.

These permissions are typically implemented as a bit mask, with one bit for each letter in the permissions string. Each cluster of three permission bits can be represented by an octal number: 000 through 111 in binary, which is 0 through 7 in decimal. For example:

The file in the example above is said to be in "mode 640."

There are actually at least three more permission bits that precede the "owner" bits. These control permission inheritance for directories (often in a vendor-specific way) and the ability of the user executing the file to temporarily assume the identity of the file's owner or group. Thus, the full "mode" string is "0640" in the example above, assuming none of the "extra" three bits are set.

You can read more about the display and modification of file permissions by running "man ls" and "man chmod" from the command line.

Access control lists

Traditional Unix file permissions are flexible, but it's still not hard to construct scenarios in which they do not offer enough control. For example, imagine trying allow a single user, Bob, to read one of your files. With traditional Unix file permissions, the only way to do it is to make a new group (say, "friends") with just two people in it, you and Bob. Then you'd change the file's group to be your new two-member group, and enable the group read permission.

That was awkward, but now imagine that you want to let a third user, Ray, write to the file, but not read it. Now you're stuck. If you put Ray into the "friends" group, he'll be able to read the file. And if you grant write access to the "friends" group, then Bob can write to the file. Since a file can only have one owner and one group, there's nothing you can do. The Unix permission system is not flexible enough to accommodate these needs.

Worse, imagine that you want to grant the ability to delete a particular file to a group of users. In traditional Unix permissions, there is no "delete" permission for a single file. The ability to delete a file is controlled by the "write" permission of the parent directory. But you want to allow just this particular file to be deleted, not all files in the same directory. The Unix permission system is not fine-grained enough to accommodate these needs.

Enter access control lists. An ACL is an ordered list of rules that control file permissions. Each rule specifies three things: an actor, an action, and whether that action is allowed or denied. A rule is also known as an ACE, or access control entry.

To determine whether or not to allow an action, the rules are considered in order. The first rule that is applicable to the current actor and action determines the decision, and no further rules are considered. If none of the rules apply, then the decision is determined by the traditional Unix file permissions.

Note that the Unix permissions are not even considered unless none of the ACEs are applicable. It's possible to have a file that "appears" to be readable to everyone, for example, but that actually allows no one to read it, or just allows a single user to read it, and so on. In effect, the Unix file permissions can be totally ignored if the access control list contains rules that apply to all potential actions and actors.

(It's actually a bit more complex in the case where multiple actions are requested simultaneously—opening a file for reading and writing, for example. But the overriding rule is the same: ACEs take precedence over Unix permissions.)

ACLs can be modified from the command line using the chmod command (run "man chmod" for details) or the Workgroup Manager application in Mac OS X Server.

For ACLs to work at all, they must be enabled for a particular volume. The Workgroup Manager application in Mac OS X 10.4 Server can do this, but that application is not included with the normal (non-server) version of Tiger. There is a command-line work-around, however. This command will enable ACLs on the boot volume.

% sudo /usr/sbin/fsaclctl -p / -e

Here's a brief demonstration of ACLs. Let's start with a file named, appropriately, "file."

% touch file
% ls -l file
-rw-------   1 john  admin  0 Mar 31 21:12 file

Right now, only the user "john" can read and write the file. Now let's revisit the earlier examples where Unix permissions fell short. With ACLs, they become trivial.

• Allow a single user, Bob, to read one of your files.

  	% chmod +a "bob allow read" file

• Let Ray write to the file, but not read it.

  	% chmod +a "ray allow write" file

• Allow a group of users (say, "staff") to delete the file.

  	% chmod +a "staff allow delete" file

You can use the new "-e" option to the "ls" command to show a file's ACL from the command line.

% ls -le file
-rw-------   1 john  admin  0 Mar 31 21:12 file
0: bob allow read
1: ray allow write
2: staff allow delete

Note again that the Unix file permissions have not changed. If Bob tries to read, Ray tries to write, or anyone in the "staff" group tries to delete the file, the Unix permissions will not be consulted at all.

The chmod command can add or remove ACEs and insert them in specific positions in the ACL. Remember, it is very significant that ACLs are ordered lists; the first rule that matches, "wins."

I mentioned the increased granularity of ACLs. This screenshot from the Workgroup Manager application in Tiger Server gives you an idea of what's possible.

Workgroup Manager ACL panel (Tiger Server only)

Remember, each ACE can allow or deny any of those actions to any user or group, and there can be an unlimited number of ACEs per file (conceptually, if not necessarily practically).

ACLs support "static inheritance." This means that the initial ACL for a newly created file may be determined by the ACLs of one or more of the parent directories. This inheritance happens once, at the time the new file is created. If the ACLs of the parents change in the future, the children won't inherit those changes.

The one-time nature of the inheritance is what makes it "static." As you might imagine, the opposite system, "dynamic inheritance," is very difficult to manage, with changes to a single directory potentially rippling through existing files and directories below it. Apple definitely made the right choice here. (As one reader pointed out, having support for both forms of inheritence might be even better. But I'd still choose "static" as the default mode.)

Tiger also does away with the 16-group limit from Panther and earlier versions of Mac OS X. Now a user can belong to any number of groups. More interestingly, groups can be nested, creating a hierarchy of groups. For example, the "staff" group could encompass all employees, while the "managers" group could be a subset of "staff," and "executives" could be a subset of "managers," and so on.

Setting an ACE like "staff allow read" will allow "staff," "managers," and "executives" all to read. Setting an ACE like "managers allow write" will allow "managers" and "executives" to write, but not members of "staff." You get the idea.

Needless to say, this is a huge windfall for anyone who has to administer a Mac OS X server. ACLs allow file permissions to more closely match the hierarchy of the organization.

ACL summary

ACLs have existed in various forms in many other operating systems over the years. In many ways, Tiger is playing catch-up here. Even classic Mac OS has some file sharing features that are beyond the capabilities of traditional Unix file permissions.

The addition of ACLs to Tiger was significantly motivated by the mismatch between Unix permissions and the Windows/Active Directory permissions model. With Tiger, Mac OS X can finally serve files to (and exist as a full peer on) a Windows network.

Before leaving this topic, I'd like readers to consider this: where, exactly, does Tiger store all of this new file permission information? Where is the "list of access controls" for each file?

Well, where are the Unix file permissions stored? They're in the file system metadata structures, of course, along with the file name, modification date, and so on. ACLs are stored in the same place. But rather than being forced to change the HFS+ volume format in an incompatible way in order to add this new metadata (requiring a complete reformat to use ACLs), the ACLs are stored in (you guessed it) extended attributes on HFS+ volumes.

(The ACL attributes are apparently masked out by the extended attribute APIs in Tiger, so you won't be able to see them with the xattr program. They also likely use the reserved "system." namespace prefix for their attribute names.)

This is just one fulfillment of the promise of that long-requested, esoteric-sounding feature, "arbitrarily extensible file system metadata." For people who couldn't see past the status quo, the ability to associate arbitrary pieces of data with any file or directory has long seemed interesting, perhaps, but largely pointless.

It appeared that Apple itself was in this camp at the dawn of the Mac OS X era. With Tiger, Apple has started to turn the ship around. Even the staunchest critic of the campaign for more, better file system metadata has to admit that it's pretty nice to get all the features of ACLs without having to reformat your disk.

Metadata sidebar

In a (mercifully) rarely read rant on the future of Mac OS X, written on December 7th of 2001, I railed against opponents of greatly enhanced file system metadata, and warned of the inevitable future that Apple may not be a part of if it doesn't wake up.

Make no mistake, the future of computing will be absolutely lousy with that hateful, performance robbing, compatibility-impairing, complexity-increasing metadata—with or without Apple. To doubt this is to deny the entire history of computing—to think that this, somehow, will play out differently than every single other technology that has come before it.

Before moving on with the article, I just want to express my relief that the anti-metadata sentiments publicly expressed by many people, and apparently shared by Apple at the dawn of the Mac OS X era, have been wisely left behind.

Steve Jobs visited Xerox PARC in 1979. In an interview in 1995, before Jobs returned to Apple, he had this to say about his first glimpse of the graphical user interface at PARC in 1979.

Within ten minutes it was obvious to me that all computers would work like this some day.

Many employees at Apple were already onboard the GUI train prior to 1979, and Jobs visited PARC largely thanks to their urging. After the visit, he saw the proverbial light, and the rest is history.

I suspect that nothing quite so dramatic or significant has happened in the case of file system metadata, but I don't think it's too far-fetched to imagine a somewhat similar situation. It's not hard to guess which side Apple employees like Dominic Giampaolo come down on, but the wildcard has always been Jobs himself. Convince him of the merits of your ideas and you're home free.

Right now, I'm not actually sure that Jobs is fully on the file system metadata bandwagon. In fact, I'm not sure he cares about—or is even aware of—the issue at the implementation level. His job is to worry about the big picture; the smart people he hires are supposed to hash out the technical details.

Two things are made clear by Tiger. First, the consensus of Jobs's technical advisers has clearly tipped in favor of the pro-metadata camp. How long ago this happened or how far it has tipped is anyone's guess. Second, Jobs is convinced enough of the merits of this approach that he is, at the very least, not actively thwarting efforts to improve this area of Mac OS X.

This may sound like damning with faint praise, but I really am glad to finally see some external sign of a turn around on what I feel is a very important issue. Metadata is the future, and I'm glad to see that Apple's decided to come along for the ride.

But wait, there's more...

Spotlight

When I first started talking about metadata in Tiger, Spotlight was probably the first thing that came to mind. Although it's the least technically daring of Tiger's metadata features in my estimation, it will, by far, be the most influential in daily use.

This is sort of what I meant earlier when I imagined that Steve Jobs spends more time thinking of the big picture and less time worrying about technical details. Spotlight is very straightforward, technologically, but it solves an important problem. Just as importantly, it does so in a way that leaves the door open for technical improvements down the road. But let's back up. What is Spotlight?

I'm going to describe Spotlight in three parts: what Spotlight does (functionality), and how it does it (implementation), and what the user interface looks like. I'll begin by describing what Spotlight does.

Spotlight functionality

Spotlight is a system service that accepts a query and returns all file system objects (files and folders) that match the query. Any property of a file system object can be tested as part of the query: file name, dates, size, etc.

The query can (blessedly) contain arbitrarily complex logic. Here's an example of a (conceptual) query.

(file name starts with "a" or file name starts with "b") and file size is greater than 1MB

If you'll allow me to digress for a moment...

Note the parentheses that delimit the first sub-clause of the query. Apple thinks so few end-users understand this concept (nested boolean logic) that it's not worth the time and effort to add support for it in any of its existing applications that include "queries" of one kind or another.

For example, I'd like to be able to make a "smart" playlist in iTunes using the following query.

(genre is "Anime Soundtrack" or genre is "Game") and my rating is greater than 2 stars

I don't think that's too much to ask. (Yes, I know it can be simulated by creating multiple playlists and referencing them from within other playlists, but that's even more complex and cumbersome than a straight-forward implementation of nested boolean logic.)

The same goes for Apple Mail and, to be fair, even Microsoft's mighty Entourage e-mail client. Yes, this is a pet peeve of mine. I think queries without support for nested boolean logic are nearly pointless.

The Spotlight API team seemed to share that opinion. Spotlight queries can be arbitrarily complex. Sadly, the default Spotlight UI does not expose these abilities, but instead falls back on the much more limited iTunes/Mail style of query building. (See screenshots in the Spotlight user interface section below.)

I think Apple has an insultingly low opinion of users' ability to understand nested boolean logic. They may not know what it's called, but the concept is intuitively understood. Even my mother has already asked me how to create a Mail rule that requires nested boolean logic: "(from A or from B) and subject contains C." I had to tell her that it's not possible. When asked why, I told her that Apple thinks few users need this ability and that it's potentially confusing. She wasn't particularly satisfied by that explanation, and neither am I.

Anyway, back to Spotlight. In addition to simply returning a list of file system objects that match a query, the Spotlight system service can also optionally send notifications every time the result set for a query changes.

Spotlight harvests information about file system objects by looking at file system metadata as well as the file data itself. This information is used to determine if a file system object should be included in a result set for a particular query.

Although I described Spotlight as a service for indexing and searching for files and folders, it's advertised as also being capable of finding Address Book contacts, Mail messages, iCal calendar events, System Preferences panels, and other seemingly non-file-related items. All of these things can appear in Spotlight search results, and each of them does have some sort of file-based presence on the system.

Mail messages are the simplest, with each message existing as a single file. System Preferences panels are actually bundles (folders, effectively), just like most applications. Address Book contacts and iCal calendar events present a bit of a mystery.

The Address Book and iCal applications use monolithic data and index files to store all their data, rather than individual files for each data item (contacts, events). Address Book uses a single, large data file (AddressBook.data) and two SearchKit index files (ABPerson.skIndexInverted and ABSubscribedPerson.skIndexInverted). The situation is similar for iCal, which uses a data and index file for each calendar, rather than for each individual event.

And yet Spotlight will find and display individual contacts and calendar events. Selecting one from a Spotlight search results list will open the appropriate application and take you right to the contact or event.

There actually are individual data files for Address Book contacts in ~/Library/Caches/com.apple.AddressBook/MetaData/, and these files are indexed by Spotlight. But instead of showing up in Spotlight search results as obscurely named data files, they appear as conceptual proxies for the data they contain. Unlike other Spotlight search results, there's no file path listed for Address Book contacts.

To pull this off, the Address Book metadata importer sets a custom kMDItemDisplayName for the files that it imports, using the contact name instead of the file name. The individual data files hidden files in ~/Library/Caches/com.apple.AddressBook/MetaData/ have a ".abcdp" extension. Opening one from anywhere, not just a Spotlight search results window, will cause the Address Book application to launch and display that particular person. Pretty sneaky.

The situation is exactly the same for iCal calendars events: more hidden, obscurely named data files that happen to be understood by the iCal application. Third-party metadata importers and applications are free to conspire to do the same thing.

It does seem like a bit of a hack—and potentially wasteful—to store both a monolithic database file and a bunch of individual data files for a given data set, but Spotlight as it currently exists is doggedly file-granular. There must be an individual file camped out in the file system somewhere for each entry in the Spotlight to metadata store.

Apple is apparently working with at least one third-party developer, Microsoft, to see what can be done in this area. Here's what a Microsoft representative had to say.

[Spotlight] won't index Entourage and that's simply due to the way that Spotlight was made versus Entourage. Entourage saves everything in a database format and we are working with Apple to see how Spotlight can index the database.

We'll see how Spotlight evolves in the future. In the meantime, it appears to be strictly file-granular, with search results for all non-Apple applications always showing up as individual files.

Finally, Spotlight can be prevented from indexing a user-configurable set of items: files, folders, or even entire volumes. All information about the selected items is deleted from the Spotlight metadata storage system, and they will not be re-indexed.

Spotlight implementation

While the functionality provided by the Spotlight system service is conceptually simple and obviously useful, the implementation—both behind the scenes and in the user interface—has its ups and downs. Let's start from the bottom and work our way up.

For Spotlight to be useful at all, it needs to harvest information about each file. This harvesting could be done at runtime in response to each query. In fact, that's how it was done in Panther and earlier versions of Mac OS X. (The exception was searching based on file contents. Panther and earlier had a background process that periodically indexed file contents.)

The HFS+ volume format stores most file metadata in a single location (the "catalog") and is therefore well suited to fast runtime searches based on file metadata. In Panther, I can search my entire boot disk (95,771 folders and over a half a million files) for, say, all files whose names contain "apple" in about 12 seconds. That's pretty fast.

Spotlight aims to be much, much faster. To do this, it has to harvest file metadata ahead of time and store it in some sort of index. When a query is run, only the index will have to be consulted. The "entire disk" (or even the entire catalog) will not have to be scanned.

Pre-created indexes allow for very fast searches, but they also preset a potential problem. It's easy for an index to get out of sync with the current state of the file system, and an index that's out of date is not very useful. In order to provide accurate results, an index must accurately reflect the state of all files "in real time."

This is quite a tall order. Obviously, a periodically running background process is out of the question. No matter how often it's run, it'd be impossible to keep it from being out of date at some point. Tiger's solution is to make metadata importing an integral part of file i/o at the kernel level.

In Tiger, file metadata is harvested by a set of metadata importer plug-ins, one for each type of file. Tiger ships with importer plug-ins for many common file types: JPEG images, text files, PDFs, etc. Developers are encouraged to write metadata importer plug-ins for their file formats. If there are multiple plug-ins for a particular file type (e.g. a generic image metadata importer and a JPEG metadata importer), the most specific plug-in wins. In other words, only one plug-in gets a shot at each file.

Each metadata importer is responsible for scanning a file and returning all of the metadata it could extract—from the file system metadata structures, the file contents, or anything else it wants to consider. The metadata is returned as a set of key/value pairs, and is added to the Spotlight index entry for the file.

Metadata importer plug-ins are stored in Spotlight folders in any of the various Library folders. As usual, the more specific locations take precedence: ~/Library/Spotlight overrides /Library/Spotlight, and so on.

Any file i/o that goes through the Tiger kernel will trigger the appropriate metadata importer. This kernel-level integration ensures that the Spotlight indexes are always up to date.

There's an important qualifier in this statement. It is only true for "any file i/o that goes through the Tiger kernel." How is it possible for file i/o not to go through the kernel? Well, it'll always go through a kernel, but perhaps not always a Tiger kernel. Here are a few sample scenarios where file system changes do not immediately become part of the Spotlight index.

• You reboot into Panther or another earlier version of Mac OS X and create or modify some files.
• An external disk drive is removed from your Mac and used with another non-Tiger computer (Mac or PC) which creates or modifies files on the disk.
• Your Mac is mounted in firewire target disk mode by a non-Tiger computer (Mac or PC) which creates or modifies files on it.

Obviously there is another, much more common example: booting into Tiger for the first time. At that point, nothing is indexed. This case is handled just like the three example cases above: Tiger will begin a background indexing process to record the un-indexed changes.

When a user first boots into Tiger, the Spotlight indexing process will initialize the Spotlight indexes. While the indexing process is running, no new Spotlight queries will be serviced. File system changes made while indexing process is running are still caught by the kernel, however. So when the initial indexing is complete, the index really will be up-to-date.

Spotlight stores index information in a set of index files on each indexed volume. These are plain files stored in a plain directory at the root of the file system. The directory name has changed over the course of Tiger's development. Currently, Spotlight index files stored in the directory /.Spotlight-V100. The file and directory names are not part of Spotlight's public interface. No one using the Spotlight APIs should know or care where or how the information is stored.

As far as I can tell, a Spotlight index on a given volume only stores information about files on that volume...although the "publish" feature of the mdutil command hints that this may not always be the case. Either way, it appears that read-only media such as CDs and DVDs are not indexed by default. Spotlight will search them the old-fashioned way, scanning their contents at runtime in response to queries.

Keep in mind that, like the storage format and location of the indexes themselves, this is an implementation detail that could change in the future. Nothing about any of the Spotlight APIs imply anything about the location or contents of the indexes.

Queries are constructed using "predicate" objects which are assembled into a tree-like data structure. A text query format is also supported by the API. It uses parentheses for nesting, C-style comparison operators (<, >, <=, >=, ==, !=) and logical operators (||, &&), a single shell-style wildcard (*), and single- and double-quoted strings using backslash () as an escape character.

There are three supported modifiers which can be applied to individual clauses within a query.

• c - The comparison is case insensitive.
• d - The comparison is diacritical insensitive.
• w - The comparison is word based, and also detects transitions from lower to upper case.

Each piece of metadata is stored in the Spotlight index using a key string. Apple has defined keys for many common attributes. These keys are defined as symbolic constants in a header file, using the "k"-prefixed Carbon naming convention for constants. They range from generic attributes like kMDItemKeywords, kMDItemTitle, and kMDItemComment to format-specific attributes like kMDItemAudioBitRate, kMDItemMaxAperture, and kMDItemLyricist. There are over 100 of these constants.

Apple encourages developers to use pre-defined attribute keys if at all possible. It makes little sense for every file format to use its own attribute key for "author," for example. A set of common attribute keys is a prerequisite for useful search results.

If a metadata importer plug-in wants to include an attribute that is not part of Apple's pre-defined list, the developer is free to define his own attribute key using the reverse-DNS syntax, with underscores replacing the dots (just like the extended attributes convention). For example, a value for the com_adobe_photoshop_useslayers key might be set by a metadata importer plugin-in created by Adobe for Photoshop files.

Here is a sample Spotlight query using the text query format described earlier.

(kMDItemTitle == "apple*"c || kMDItemTitle == "mac*"c) && kMDItemFSSize >= 5000

This query would match any file whose title starts with "apple" or "mac" (matching case-insensitively) and whose file size is greater than or equal to 5,000 bytes.

Tiger ships with a nice collection of command-line tools for Spotlight, all of which share the "md" prefix (which stands for "metadata," in case you haven't guessed already). The mdfind command accepts text queries like the one shown above and prints out all the matching files. The mdls command will list all the attribute names and values in the Spotlight metadata store for a particular file. The mdimport command forces a file to be (re)indexed by Spotlight. The mdcheckschema command is used to validate the configuration of an individual metadata importer plug-in. The mdutil command, which must be run as root, is used to manage the Spotlight metadata stores. These tools are useful for developers creating metadata importer plug-ins, or even for users who are just curious.

Spotlight user interface

Tiger includes a small magnifying glass icon in the upper-right corner of the menu bar. Clicking it reveals a Spotlight search field.

Spotlight menu

This menu can also be activated with a user-configurable keyboard shortcut. The default is command-space, an obvious nod to LaunchBar and Quicksilver, two third-party quick-search applications.

Typing in the search field runs a Spotlight query in real-time as you type. It works a lot like the search field in iTunes, and it's nearly as fast. It slows down substantially when searching unindexed volumes like read-only optical disks, but the query results arrive as they become available. The UI never blocks waiting for all the results to arrive. It even defers the loading of icons for the search results if that will get the results list to appear sooner. The perceived latency is usually very good.

Spotlight menu with search results

The trade-off is that the results list tends to jump around as new results arrive. This can make navigating and selecting an item from the menu much more difficult than it should be. A better solution would be to stop searching once the user starts to make a selection, or to start adding new results only to the very end of the list. Anyway, it's still better than having the entire UI block until all results are in.

The text typed into the search field is implicitly matched against a large set of attributes. Again, this is like iTunes where a search string is implicitly matched against artist, album, track name, and so on.

The result set is trimmed to show only the top results in each category. The number and order of these categories can be customized using the Spotlight preference pane in System Preferences. The search results drop-down menu can be navigated with the arrow keys. Hitting return will open the selected item. The top choice in the drop-down opens a new window that displays a larger set of results.

Spotlight search results window

From this window, the user can refine the search, change the sorting or grouping, and display all results in each category. Expanded information is available for all file types, with previews available for most.

Spotlight window showing image thumbnails

Some categories have more elaborate previews. For example, images can be displayed in a full-screen slide show. During the slide show, each image can be zoomed to fill the screen, or copied into the user's iPhoto library with a single click (complete with a genie-like animation). There's also a "contact sheet" effect that tiles all the images in a full-screen grid of thumbnails.

Spotlight search fields also appear in open/save dialog boxes. I suspect this location may get used even more often than the Spotlight menu bar search field, once users discover it.

Spotlight search fields also appear in an open/save dialog boxes

Notice the "TextEditScreenSnapz001.pdf" file in the results list. When I first took this screenshot (with the excellent screen capture utility SnapzPro X) I forgot to specify that I wanted to use the PNG file format. Snapz used the default PDF file format instead. When I corrected my mistake and took the screenshot again, I noticed that the PDF file from my original screenshot was in the results list.

Using the handy mdls command mentioned earlier, I discovered the following piece of metadata associated with the "TextEditScreenSnapz001.pdf" file.

kMDItemEncodingApplications = ("Mac OS X 10.4 Quartz PDFContext")

It appears as that every single PDF screenshot I take with SnapzPro will end up in the results list for the query string "quartz." I'm not sure if this is a bug or a feature. It's usually handy when search strings are matched against many pieces of metadata simultaneously, but perhaps there is such a thing as being too broad.

Selecting "Find..." from the Finder's "File" menu (or typing another user-configurable keyboard shortcut, command-option-space by default) brings up an "advanced" Spotlight search window that includes a field for a search string (treated like the same string typed into the Spotlight menu bar drop-down search field) and any number of additional search conditions.

I put "advanced" in quotes above because the full power of the Spotlight query system is not exposed in this interface, or in any other Apple-provided interface that I have found. Instead, all conditions are "AND"ed together in a single, flat chain. Apple's insistence on the average user's inability to understand (and lack of desire to perform) complex queries strikes again!

Any search performed in this window can be saved as a "smart folder." This "folder" is actually nothing more than a plain file with a ".savedSearch" file name extension. The file contains an XML serialization of the Spotlight query.

The Finder and open/save dialog boxes in Tiger treat these files specially, displaying them as if they are read-only folders containing the items matched by their associated queries. Smart folders can be placed anywhere a file can: in the sidebar of a Finder browser window, on the desktop, in other folders, even in the Dock. Unfortunately, the Dock does not treat smart folders as folders. Instead, they behave like the plain files that they actually are. Boo.

While a smart folder is open in the Finder, Spotlight's notification functionality keeps its contents up-to-date in real time. Yes, it really, truly works in real time, as this movie demonstrates. On the right is a smart folder based on the query "boogawooga." It currently matches zero files (imagine that). On the left is a TextEdit document. Now it's magic time...

It's amazing what a few kernel hooks will get you, eh?

Overall, however, the Spotlight user interface belies the strength of the service itself. The Spotlight UI is odd, clumsy, limiting, and often downright user-hostile.

To begin with, Spotlight windows do not appear to belong to any application. There is a precedent for system-service-type features being run by non-obvious applications in Mac OS X. (Exposé is run by the Dock, for instance.) But Spotlight is different in that it has its own windows: the windows showing "all results" for a search initiated from the menu bar search field, and the windows that appear when you press (by default) command-option-space to begin a new search in a window instead of the menu bar.

These windows exist in a no-man's land. There is no Dock icon to click on to bring all Spotlight windows forward. Spotlight windows do not appear in any application's "Window" menu. If you trigger the "Application Windows" mode of Exposé and cycle through all the sets of windows using the tab key, you won't see your Spotlight windows there either.

There are only two ways to get at a Spotlight window. You either find it by digging through or hiding other windows, or you trigger the "All Windows" mode of Exposé and pick it out of the complete set of windows.

At least the Finder implements (and therefore "owns") its own Spotlight search windows, triggered by selecting "Find..." from the "File" menu or by typing command-f. Unfortunately, these windows are quite maddening in their own right.

First, they exhibit the now-traditional Mac OS X Finder window schizophrenia, changing from brushed-metal-with-a-sidebar to a "standard" window with a single click. While this transformation is meant to distinguish "browser" windows from "folder" windows under normal circumstances (and deals a mortal blow to the spatial consistency of the Mac OS X Finder in the process), it is absolutely nonsensical in the context of Spotlight.

Search windows in the Finder are inherently "browser" windows. All the transformation ability does is create yet another way for window frames, controls, and contents to jump around the screen without being explicitly asked to do so. I almost expect the search form controls to actively try to squirm away from my cursor at this point.

And how about those controls? Both incarnations of the search window (Jekyll and Hyde, you decide which is which) have a text search field. In the metal version, it's an actual search field widget (the round-ended text field) up in the toolbar area. In the standard window, it's a rectangular text field above the content area of the window.

The Finder's Spotlight "Find..." window: the metal version...

...and the non-metal version

In both cases, there are two additional search criteria represented by pop-up menus for "Kind" and "Last Opened" filters. You'd better get used to those two conditions because you're going to be seeing them a lot. How much is a lot? How about every single time you open a new search window in the Finder.

Not enough? Try opening a smart folder. If it's metal, and if you removed those two conditions earlier (using the round minus "-" buttons), they'll be gone. But woe be unto you if you foolishly click the toolbar widget and transform the window into it's non-metal variant. Behold, your good friends "Kind" and "Last Opened" have reappeared! Now try changing back to a metal window—hey, they've decided to stick around! Quick, just close the window and be done with it! Now open it again...hm, looks like we're back to square one.

But wait, there's more! Open a smart folder, switch it from metal to standard, then close it. When you reopen it, it will not be metal (a miracle!) but it will display everyone's favorite search criteria, "Kind" and "Last Opened," and the text search field will have the keyboard focus! Yes, that's right, open a smart folder and type a character accidentally to completely change your "saved" search. Whoops.

Sure, the change doesn't become saved immediately and you will be prompted to confirm the change (via a sheet) when you close the window, but come on. What is the point of a "saved search" that can be so easily modified, even if transiently?

This is what I'm talking about when I say the Spotlight UI exhibits "user-hostile" behavior. It's like the middle finger (have some "Kind" and "Last Opened," loser) combined with a booby trap (editable search criteria and default input focus on the text field in smart folders).

Here's some video of the Finder doing what it does best: confounding user expectations and absolutely hosing any semblance of consistency and statefulness. The movie shows the smart folder from the earlier video (the one with the "boogawooga" search string and no other search conditions) being opened and closed in both metal and non-metal modes. While watching, just try and guess how the window will look when it's re-opened after each of the demonstrated actions. You may need to step through the movie slowly to get a full grasp of the insanity.

Under what set of circumstances does this get to ship? I would love to see the "design document" for Spotlight's integration into the Finder's interface, if such a thing even exists. (I highly doubt that it does.) I'm tempted to say this is par for the course when it comes to Finder windows in Mac OS X. But really, this is way beyond the Finder's standard level of user abuse.

I tried to film the user-hostile behavior of the "Find..." window, but it so thoroughly neglects to retain state that (in addition to eternally including the "Kind" and "Last Opened" conditions) it insists on opening in the same location and at the same size every single time—a size that is a bit too large to comfortably fit in a movie. No matter, I'm sure you'll become intimately familiar with it on your own time. As the kids say, "gg Apple."

Maybe you think I'm making too much of this, but just think about the consequences of having to repeatedly modify these stubborn windows every single time they're used in anything other than their default conditions. Don't like to use a metal search window in the Finder? You can look forward to clicking the toolbar widget every single time you search. Don't like the default size or location of the search window? Have fun moving and resizing it every single time (or looking for some sort of plist hack to set some new defaults). Maybe you don't use extra search criteria beyond just the text field? Ha ha, you get "Kind" and "Last Opened" and you'd better like them! Maybe you use some different criteria more often? Tough.

Still, these are little things, right? Let's seek some wisdom from Steve Jobs himself.

You know, I've been thinking about it. How many people are going to be using the Macintosh? A million? No, more than that. In a few years, I bet five million people will be booting up their Macintoshes at least once a day.

Well, let's say you can shave 10 seconds off of the boot time. Multiply that by five million users and thats 50 million seconds, every single day. Over a year, that's probably dozens of lifetimes. So if you make it boot ten seconds faster, you've saved a dozen lives. That's really worth it, don't you think?

Don't tell me, Steve, tell the Spotlight UI team.

Spotlight analysis and potential

First and foremost, Spotlight is an excellent implementation of a simple, effective design for a system-wide file system search. It's very fast, very capable, and always as up-to-date as is technically possible. It is so much better in all respects than the Panther file system search feature that a comparison is not even worthwhile.

Comparisons to LaunchBar and Quicksilver seem inevitable, but Spotlight is really in a different category. Those two third-party applications are content to confine themselves to smaller subsets of the file system, whereas Spotlight is meant to be as broad as possible.

Spotlight is also entirely file-centric, whereas Quicksilver in particular deals with many more kinds of things (web searches, dictionary look-ups, bookmarks, active processes) and can do a lot more with them (move, copy, rename, delete, activate, kill, get info, etc.) I actually expect both LaunchBar and Quicksilver to be strengthened as products by leveraging Spotlight's APIs where appropriate (system-wide file searches), while retaining their own unique features.

I'm a bit disappointed in the implementation of smart folders. While Apple was mucking around in the kernel on behalf of Spotlight, couldn't they also have implemented smart folders more transparently? Picture this.

A smart folder could be a normal directory that is specially tagged using an extended attribute (in the "system." namespace, masked-out just like the extended attributes used for ACLs). The actual Spotlight query for the smart folder would also be saved in an extended attribute. The contents of the smart folder would be generated on the fly in response to file i/o system calls (opendir(), readdir(), etc.) and would appear to be a series of read-only hard links to the actual files.

Suddenly, applications no longer have to be updated to be "smart folder savvy." Instead, smart folders just naturally work as expected everywhere, even from the command line. It's a beautiful thing.

Apple hasn't even provided this level of transparency for aliases (a shame), but Spotlight proved their willingness to fiddle with the kernel on behalf of a vastly improved user experience. I think it's a natural extension of this value system to implement smart folders as I've described.

By spending several pages talking about file system metadata in Tiger before even mentioning Spotlight, I was trying to prove a point. That point is this: Spotlight is a search and indexing service—a very good one, mind you, but nothing more than that. It has almost nothing to do with arbitrarily extensible file system metadata as discussed, debated, and petitioned for during Mac OS X's development, and as demonstrated in operating systems like BeOS and file systems like BFS.

Yes, Spotlight extracts, stores, and indexes information about file system objects. Yes, this information is properly called file metadata. But this information is extracted from the file contents and traditional file system metadata fields (file name, dates, size, etc.) and is stored in external plain-file indexes.

The only way actual, arbitrarily extensible file system metadata is involved at all is if an application chooses to write extended attributes when it saves a file, and then a Spotlight metadata importer plug-in reads these extended attributes and passes their values off to Spotlight for storage in its index files. At the time of Tiger's launch, no existing applications or metadata importer plug-ins do this.

Spotlight merely extracts, stores, and indexes file metadata. It does not and cannot be used to add arbitrary metadata to files. It can read a file and add metadata to the Spotlight index on behalf of that file, but the metadata is not "physically" attached to the file itself.

Why is all of this significant? To answer that, it helps to understand what has motivated the call for arbitrarily extensible metadata in Mac OS X.

The ability to associate any piece of information with any file system object provides an unlimited number of new ways to organize files. The "default" organization is based on file name and file path—two pieces of metadata that are so taken for granted that they don't even seem like metadata most of the time. In the minds of most users, there is simply a file hierarchy. In reality (and especially in file systems like HFS+), file path and file name are just two more pieces of file system metadata that happen to be treated specially in some circumstances.

The promise of arbitrarily extensible metadata is that new structures can be defined by the user by adding metadata. Imagine tagging certain files as being part of "The Brinkman Project" or "Urgent" or "Overdue." Now imagine making a series of smart folders containing "all overdue Brinkman files" or "all urgent video clips created this month."

You may be thinking that Spotlight can already do this, but Spotlight is limited by the metadata that can be extracted by the importer plug-ins for each file format. Microsoft Word may have a "Project" metadata field in its file format, but plain text files certainly do not. If you want to tag a heterogeneous set of files as belonging to a single project, your only recourse is the file hierarchy (i.e., put them all in the same folder).

The same goes for any other piece of metadata that isn't guaranteed to exist in all (or perhaps any) file formats: urgency, due date, subject matter, client, etc. If you want to organize files based on such an attribute, Spotlight is not going to help you much.

What's needed to really break free of the file system hierarchy is a way for the user—not applications, not file formats, but the user—to be able to define and apply arbitrary metadata to any file. Tiger's extended attribute APIs provide this functionality, but it is currently untapped.

To imagine how this might work in the future, consider this scenario.

• A user, Joe, creates an image in Photoshop. When Photoshop saves the file, it attaches information about the file using extended attributes: the number of layers, the color space, the resolution, etc. In cases where there is an Apple-defined attribute key (e.g., kMDItemPixelWidth), it is used. Otherwise, keys that begin with com_adobe_photoshop_ are used.
• Spotlight, being conveniently hooked into the kernel, detects the creation of this file and triggers not one metadata importer plug-in, but two. The first is a generic importer plug-in that runs on every file. It reads all of the standard file system metadata (file name, dates, size, etc.) plusall of the file's extended attributes. All that information is added to the Spotlight index.The second plug-in is the one associated with Photoshop files. It understands the Photoshop file format and it extracts any information that can be derived from the file contents themselves. Ideally, there is nothing for this importer to do since a well-behaved application will write all interesting information into a file's extended attributes when it is saved. But format-specific importers are still needed for applications that have not been updated to do so.
• Joe repeats this process for several more files of various types. Each time a file is created, the creating application attaches any interesting metadata using extended attributes, and Spotlight extracts and indexes this metadata.
• All of these files are all part of the Brinkman Project. To indicate this, Joe selects them all in the Finder and attaches a new piece of metadata to them ("Project: Brinkman"). If the "Project" attribute is not already defined, Joe adds it so that it will appears in the pop-up menu of possible attributes in the future.
• Ever vigilant in the kernel, Spotlight notices that these files have been modified and dutifully re-indexes them, now picking up the new metadata attribute of "Project: Brinkman."
• Joe repeats this process as necessary, tagging files as "overdue" or "urgent," applying due dates or workflow stages such as "draft" or "final," and so on.
• In each Finder window, Joe can choose to display any one of these pieces of metadata in addition to the usual file name, dates, size, etc. Additionally, any one of the attributes can be assigned a color, allowing red to mean "urgent," brown to indicate a "draft," and so on.
• Finally, Joe is now free to define smart folders like those described earlier: "all overdue Brinkman files" or "all urgent video clips created this month" or "all draft images for the Rand Corporation."

That's the metadata-powered future that I envision, and that BeOS made a bold grab for. Spotlight and extended attributes bring Mac OS X very close, but only potentially. It's still not clear how committed Apple is to the concept of extended attributes. As I hope I've made clear by now, they are an essential part of this vision.

Speaking of BeOS, there's another big difference between Tiger's metadata technologies and Be's implementation. The Be file system, BFS, didn't just store extended attributes; it also indexed them. The file system itself stored and maintained the metadata indexes.

In Tiger, HFS+ merely stores extended attributes. Spotlight is responsible for building and maintaining the indexes. Both the HFS+ file system driver and Spotlight are hooked into the kernel, so the overall effect is similar, but there are some significant distinctions.

BFS's use of file-system-resident metadata indexes provides more flexibility in data layout. In BFS, a small file with a few pieces of metadata could be read from a single run of contiguous disk blocks. With Tiger's approach, using external index files to store metadata extracted largely from the internal structures of particular file formats, the disk head has to visit at least three locations on the disk: the catalog file (to get the "normal" file system metadata like file name, dates, and size), the file itself (to get the data), and the Spotlight index file (to get the "extra" metadata).

This example ignores caching for the sake of simplicity, and particular implementations of each approach may vary. But in general, any time the disk head has to move, it is very, very bad for performance. The "integrated" approach pioneered by BFS has the potential to be much faster and more efficient.

The second thing in favor of the BFS approach is reliability. It's a simple fact of life that file system driver code is held to a higher standard than almost any other code—even kernel code. Certainly application code can afford to be much more lax. If an application crashes, it's annoying, but an application can always be relaunched. If the kernel crashes, well, that's pretty bad, but at least you can reboot.

But if a file system driver has a bug, it could potentially corrupt or destroy all of your data. No relaunch or reboot is going to fix that. And while application and kernel bugs can also corrupt or destroy files, a file system driver bug is much more likely to do so.

As a consequence, file system drivers are among the most reliable pieces of software in any operating system. Putting metadata indexing into the file system itself ensures that this code will be as reliable as the rest of the driver code. Users simply will not tolerate buggy file system drivers, period. One only has to witness the torture tests that BFS endured during development to see the how the reliability constraints of being in the file system driver can benefit a piece of code.

Finally, it simply seems appropriate for the code to be in the file system driver and the indexes to be integrated into the file system itself. This eliminates the need for two separate services to synchronize with each other in the kernel (e.g., the HFS+ driver and the Spotlight metadata importers).

That said, there are several advantages to Tiger's approach as well. While the system used in BeOS only worked with BFS, Spotlight indexes (being plain files) can be created anywhere. Tiger's extended attribute APIs also degrade gracefully on file systems that lack native support for them. (Well, I'm not sure how "graceful" dot-underscore files are, but the point is that it technically works.) And thanks to its use of external indexes, the Spotlight approach also has the potential ability to index read-only media (even if it doesn't do so in version 10.4.0).

I'd like to see a hybrid approach in Mac OS X's future, with "native" Mac OS X volumes using a new, modern, high-performance volume format created by Apple with the ability to store and index arbitrary file system metadata in the BFS style, plus the current external index approach for use with lesser file systems, read-only media, and the like. This would be the best of both worlds.

Here's something else to note about "Joe's" metadata-powered workflow described above. The first bullet point in the scenario describes Photoshop storing metadata in extended attributes using Apple-defined keys like kMDItemPixelWidth as well as keys prefixed with com_adobe_photoshop_. As I've tried to emphasize thus far, extended attributes and Spotlight are really orthogonal technologies. And yet here I am apparently mixing them, creating extended attributes using predefined Spotlight constants and reverse-DNS Spotlight-style custom attribute names.

This is cross-technology event is made possible because, as some readers may have noticed already, Spotlight metadata indexes and extended attributes share the same naming convention. I find it highly unlikely that this is a mere coincidence.

The really beautiful thing about all of the fantasy scenarios described above is that there's nothing about any of the public APIs in Tiger that precludes any of them. Perhaps I am being unreasonably optimistic, given Apple's recent history in this area, about the foreshadowing of, and prospects for, my fantasy scenarios. But at the very least, the sign of a great API is that it does not unnecessarily constrain future initiatives.

The simple, straightforward APIs for Spotlight and extended attributes in Tiger leave many doors open. And although I am disappointed in the smart folder implementation, keep in mind that there are no public APIs that deal with them. Smart folders are essentially a one-off creation of the Finder that a few other Apple applications have been updated to understand. So the door's still open on them too.

(Unfortunately, third-party applications can and will learn to read the ".savedSearch" smart folder files, despite the lack of a public specification for them. Any changes to the smart folder implementation that Apple makes down the line will have to deal with the inertia created by the current implementation, documented or not.)

Spotlight's menu bar and "advanced" search user interfaces are decidedly quirky, but they shine compared to the Finder's clumsy attempt to integrate the service. Happily, there's nothing to stop a third-party developer from creating a better front-end to Spotlight. Remember, the Finder and even the "native" Spotlight UI itself (the menu bar and search window) are not doing anything special. They're using the same public Spotlight APIs that any application can use.

That's why I'm excited about Spotlight as it exists in Tiger, warts and all. While I have little hope that the Finder will straighten up and fly right any time soon, I'm absolutely sure that many of the applications I use every day will soon integrate Spotlight in much more interesting and useful ways. Finally, the minimalist, abstract public interface to Spotlight's functionality leaves Apple with nearly unlimited options when it comes to future implementations. My fingers are crossed.

File types revisited

File typing has been a hot topic in Mac OS X ever since file name extensions became mandatory in Apple's interface guidelines in 2001. Philosophical issues aside, there are real, practical problems with file name extensions as a means to store and identify file type information. Let's even ignore storage issue entirely and focus just on identifying a file's type.

The first problem with file name extensions as a means to identify a file's type is that they are not unique. File name extensions are used on many operating systems, and there is no universal arbiter for which extensions indicate which file types.

Maybe "universal" uniqueness is too high a standard. But even within a single operating system, many file name extensions are overloaded. Even ".doc", which most people know as the file name extension for Microsoft Word documents, is used by at least four other applications: FrameMaker, WordStar, WordPerfect, and DisplayWrite. Clearly, a file's type cannot be conclusively and uniquely identified based on its file name extension.

This problem alone is sufficient to disqualify file name extensions as a "correct" (let alone "good") system for file type identification. But ignoring even this failure, there is another inadequacy. File name extensions have historically been opaque; the entire extension is a single string with no predefined internal structure. This precludes any sort of namespace or inheritance system.

Apple has been more expressive than most vendors with its file name extensions. Smart folders have a ".savedSearch" extension; text clippings have a ".textClipping" extension; frameworks have a ".framework" extension. In the Windows world, these extensions would probably be something like ".smf," ".txc," and ".fwk," respectively. (Despite the long-dead three-character limit on file name extensions in Windows, Microsoft continues to favor "compatible" file name extensions—and file names, for that matter.) But either way, extensions remain imprecise and inflexible.

There have been many other file typing systems over the years. Classic Mac OS used a combination of two 32-bit integers, usually expressed as four-character strings, to identify file types. Apple maintained a registry of type and creator codes. Since no other platform used them, they were guaranteed to be unique among all respectable Mac applications. But type and creator codes are even less expressive than file name extensions, especially the verbose ones used in Mac OS X. (The sane storage location for type and creator codes more than made up for this, however.)

More recently, MIME, a data typing system originally designed for use with e-mail, has spread to other domains, most notably the web. MIME types are hierarchical text strings with well defined rules controlling their format. While a JPEG image file might have a ".jpg" or ".jpeg" file name extension and a "JPEG" classic Mac OS type code, the MIME type would be "image/jpeg" which indicates that a JPEG is a subtype of "image." There are only a few official primary types (image, video, audio, text, application, etc.) but there are several mechanisms for vendor-specific ("application/vnd.ms-excel") and ad-hoc ("x-world/x-3dmf") expansion.

BeOS, pioneer of all things metadata related that it was, chose to use MIME as its native file typing system. It was superior to both the classic Mac OS and Windows approaches, storing file type metadata in its own dedicated location like classic Mac OS, but also supporting a file type hierarchy and expansion mechanism.

BeOS didn't live long enough to find this out the hard way, but MIME has its weaknesses as a file typing system as well. MIME's confinement to a two-level hierarchy is very limiting. Sure, all image MIME types are under "image/...", but what about the vendor who creates a new image format? New subtypes under "image/..." or any other top-level type must be "registered with, and approved by IANA" according to the RFC. The only other option is to use the ad hoc "x-" prefix which offers no guarantees of uniqueness.

Now imagine that a vendor invents what can only be reasonably considered a new top-level type. Actually, it doesn't take much imagination because there are existing file formats that are already beyond the set of top-level MIME types. MIME defines top-level types for video, audio, and images, but nothing that completely covers something like, say, a QuickTime movie file which can contain video, audio, and images all in the same file.

MIME does offer the same "x-" prefix for ad hoc top-level types, but again there is no guarantee of uniqueness. There's also the "application/vnd.*" subtype, as used in "application/vnd.ms-excel", for example. But even that lacks a suitable system for uniqueness. (Microsoft gets vnd.ms-* then? Can Apple use vnd.apple-*? How about vnd.apple.*? Not reassuring...)

Finally, all of these work-arounds bypass the intended hierarchy of MIME. Anything under an "x-" prefix or buried in "application/vnd.*" will not be caught by a MIME wildcard like "image/*", which is meant to catch "all images." Such a match will miss any image that is vendor-specific or otherwise not part of the (small) "officially approved" list of image MIME types maintained by IANA.

So, great, we've just eliminated file name extensions, classic Mac OS type and creator codes, and MIME types as suitable file typing systems. What's an OS vendor to do?

As part of its ongoing (I hope) file metadata penance, Apple has taken another look at file typing in Mac OS X. The situation in Panther and earlier versions of OS X was pretty grim. All of the different file typing systems described above were used in one place or another. But if forced to name a single, "official" file typing system for Mac OS X, it'd have to be filename extensions.

Filename extensions were the only file typing system all OS X applications were required (by the Apple human interface guidelines) to use. Applications could decide to write type and creator codes, and many did, at least initially. As time wore on, fewer and fewer applications bothered.

MIME types were mapped to file name extensions, and sometimes type and creator codes, by applications that needed to deal with MIME-encapsulated or tagged data (e.g., Safari and Mail). But without support for extended attributes, there was no place to actually store MIME types along with the actual files.

There were APIs for dealing with all three kinds of file type data. Despite the de-facto standardization on file name extensions as the one, true (crappy) file typing system in Panther and earlier, different APIs required different file type arguments. Especially during the early days of Mac OS X, the Carbon APIs tended to work with type and creator codes, while the Cocoa APIs dealt mostly with file name extensions. By the time Panther rolled around, most APIs had learned the tricks of their siblings.

Panther actually introduced a fourth, more generic "data typing" system for use with "pasteboard" ("clipboard" to long-time Mac users) data. There was little fanfare and even less documentation. Only developers who actually had to use the APIs were likely to take notice, and to them it probably didn't look much like it had anything to do with file typing.

Well, in Tiger, that oddball pasteboard data typing system has flowered into something much more important: the new One, True File Typing System for Mac OS X. And unlike file name extensions, it doesn't suck. In fact, it's superior to all of the file typing systems described above. Finally, a solution befitting "the world's most advanced operating system."

Apple's new system uses what are called "Uniform Type Identifiers" (with the medically unfortunate abbreviation "UTI"), and here's how they work.

Uniform Type Identifiers

The first thing to understand about UTIs is that, in Apple's words, they "uniquely identify a class of entities considered to have a 'type.'" That's about as broad a description as you can imagine, but in plain English it means that UTIs describe a lot more than just "file formats." They can also describe data (e.g., in the pasteboard or in a stream), structured groups of files and folders (e.g., Mac OS X bundles), symbolic links, hard links, even entire disk volumes. It really is as Apple describes: if it exists as an identifiably typed entity on the system, it's fair game for a UTI.

UTIs set out to solve all of the problems of file name extensions, type and creator codes, and MIME types.

• Uniqueness - As a brand new system for data typing, UTIs are unencumbered by past mistakes. They are defined as being unique, and there is a system for ensuring that they remain so.
• Expressiveness - UTIs have no restrictions on length and have associated human-readable (and optionally localized) descriptions.
• Extensibility - UTIs support a well-defined system for adding new types easily and safely, both with and without the blessing of the organizing body (Apple, in this case).
• Taxonomy - Selecting a subset of data types (e.g., "all images") is straightforward and does not exclude any vendor-specific or ad-hoc UTIs.

Mmmm, I love the smell of ambition in the morning. Smells like... hierarchy.

UTIs exist in a tree-like structure where each node is said to "conform to" its parent nodes. All JPEGs are image files, all images files are data files, and so on. MIME tried to create a similar sort of hierarchy, with all images under the "image/..." prefix. But as covered earlier, this breaks down quite badly when vendor-specific file formats are thrown into the mix, and the number and scope of top-level types isn't very expressive either.

From my perspective, the primary breakthrough—the "big idea," if you will—of UTIs is that they separate the hierarchy within each individual UTI from the hierarchy of the UTIs with respect to each other. Here are the basics.

UTIs are text strings that can contain numbers, letters, hyphens, and periods, plus any Unicode characters above the ASCII range. Standard types have a "public." prefix. These are also called "public identifiers," and only Apple can define them. UTIs created by anyone other than Apple must use (surprise) the reverse-DNS naming style. (e.g., com.adobe.pdf)

Each UTI can "inherit from" or "conform" to any other UTI. Below is a sample Uniform Type Identifier hierarchy.

Here we can see that HTML files (public.html) and plain text files (public.plain-text) are both kinds of text files (public.text). In turn, all text files are data files (public.data), and so on. Again, note that the hierarchy within the UTI names has absolutely nothing to do with where they are in the tree. The UTI com.mycorp.myspecialtext is a kind of plain text file, but the UTI name itself doesn't begin with "public.", let alone public.plain-text.

Also keep in mind that public.data is not the top of the UTI tree by any means, nor is there one single root. Any UTI that does not inherit from another UTI can be considered a root.

UTIs can inherit from multiple parents as well, as in the case of application packages (the specially structured folders that appear as single application icons in Mac OS X). Application packages are both "bundles" and "packages."

Applications declare UTIs in their Info.plist (XML property list) files inside their application bundles. There are two kinds of declarations, "exported" and "imported."

Exported UTIs are made available to the entire system. An application that has its own file format should have an exported UTI declaration describing that format.

Imported UTIs declare UTIs that are not "owned" by the application, but that need to be known to the system in order for the application to do its work. For example, imagine an application that does some sort of post-processing on Adobe Photoshop files. Such an application should continue to work even if Photoshop is not installed on the system.

In order for Mac OS X to, say, handle drag and drop properly for this application, it must understand that the file being dragged is an Adobe Photoshop file. But if Photoshop (which has an exported UTI declaration for the Photoshop file format, com.adobe.photoshop) is not installed, how will the system know what a Photoshop file is? To solve this problem, the Photoshop file processor application includes an imported UTI declaration for com.adobe.photoshop, essentially "pre-declaring" the existence and nature of the Photoshop file UTI, even if Photoshop isn't installed.

In the case where an exported UTI declaration conflicts with an imported UTI declaration, the exported declaration takes precedence. This makes sense, since the exported declaration is made by the application that "owns" the UTI. Once it is installed, the imported UTI declaration is no longer needed anyway.

A UTI declaration includes the following information.

• The UTI string itself.
• A list of UTIs that it inherits from.
• A list of alternate type identifiers that are equivalent to this UTI (file name extensions, MIME types, etc.)
• The path to an icon file in the application bundle used to display files of this type.
• An optionally localized human-readable description.
• The URL of a document describing the UTI.

This list of "equivalent" types is used to provide conversion APIs from one type to another. As you might imagine, it's entirely possible that any given file name extension, classic Mac OS type code, or MIME type will resolve to more than one possible UTI. The conversion API accepts "hints" to help pick one (e.g., only consider subtypes of public.data because I got this file name extension from a file, not a folder or other type of entity). Without any hints, the API prefers public identifiers over others.

In the case where there is no suitable UTI for an item, a dynamically generated UTI is used. These begin with a "dyn." prefix and exist to ensure that everything on the system has some sort of unique UTI, even if it is only transiently meaningful.

Developers are encouraged to inherit from public identifiers unless the proprietary UTI they want to inherit from is defined in the same application (or, presumably, in another application made by the same company).

Apple has published a list of almost 100 predefined UTIs that cover a huge range of entities. Here are just a few examples.

As you can see, nothing is safe from UTIs. Apple also includes imported UTI declaration for common file formats such as Adobe PDF, Microsoft Word and Excel, etc.

In what must be the ultimate act of dominance, UTI declarations actually use UTIs to describe the other data typing systems.

As a man in an orange Bandicoot suit once said, "Booya, Grandma! Booya!"

File typing summary

It's simple. When it comes to uniquely identifying data types, MIME types, type and creator codes, and file name extensions all suck, and UTIs rule. While MIME types and type and creator codes may be better data typing systems than file name extensions (especially when it comes to storage location), they suffer from many of the same problems. Just look at some of the examples above and note how many different type codes and MIME types mean the same thing: 'MPG ', 'MPEG' video/mpg, video/mpeg, video/x-mpg, video/x-mpeg. It's not just file name extensions that suffer from uniqueness problems.

UTIs wipe the slate clean and bring many new abilities to the table. The only obvious way that UTIs could be improved is for them to become an open standard rather than a system administered entirely by Apple and used only in Mac OS X. But really, those things would just be icing on the cake. So long as values from any other data typing system can be mapped to UTIs, which typing system other OSes use is of less concern. One would hope that they'll catch up eventually, but in the meantime Mac OS X Tiger has a powerful, extensible data typing system that's well beyond its predecessors.

A great standard is nothing if it is not used, of course. I expect all Mac OS X APIs to slowly migrate towards using UTIs as the preferred means of data typing. In fact, remember when I mentioned that Spotlight uses one metadata importer plug-in for each file type? Well, the "file types" Spotlight uses to match each file to a particular metadata importer plug-in are actually UTIs. Each importer must indicate a single UTI that it is able to index. The UTI hierarchy comes into play when Spotlight chooses the "most specific" metadata importer for each file based on how closely the file's UTI is matched by each of the (possibly several) applicable importers.

Tiger already includes APIs for converting older data types to and from UTIs. Developers can begin changing their application to use UTIs exclusively when making internal decisions about file types. Apple appears to be encouraging this.

Unfortunately, UTIs exist only as a concept internal to applications. Apple has chosen not to take the obvious next step and move the recommended external representation of file type away from file name extensions. File name extensions are, by far, the worst of the file typing systems, but they're also the most commonly used. I've never objected to (and have actually supported) the use of and support for file name extensions in Mac OS X. The terrible mistake Apple made was to bless file name extensions as the "standard" file typing system in Mac OS X.

With UTIs, Apple has effectively reversed at least half of that decision. Internally (in applications and the OS itself), UTIs are now the recommended standard, not just file typing, but for all "typing" in general.

But externally, file name extensions are still the only mandatory file typing system according to Apple's interface guidelines. That's a shame because, with the advent of extended attributes, Tiger has the ability to store UTIs along with every file as the "official" file type identifier. Apple has simply chosen not to recommend that practice.

(Interestingly, type and creator codes are already (redundantly?) stored this way in Tiger. Assigning them to a file results in a new extended attribute key, com.apple.FinderInfo, with a value that is the concatenation of the type and creator codes. This is done transparently whenever a type or creator code is assigned to a file.)

With the addition of the conversion APIs, all the pieces are already in place for UTIs to be officially blessed as the recommended external file typing system in Mac OS X. File name extensions could then safely be relegated to "optional, but on by default" status. Finally, the file name would be entirely back under the user's control, as it should have been from the start.

All of that remains a dream, or at least my dream. In the meantime, it's nice to finally see some sort of movement on this front in Tiger. It was long overdue. What has been done in Tiger has been done well. Although it's less than I wanted, it's better than I expected. I only hope the trend continues.

Metadata summary

And so we finally arrive at the end of the ridiculously long metadata portion of this article. It may seem like I gave disproportionate attention to this topic, but remember that I have barely covered it at all since Mac OS X 10.1 in 2001—mostly because little changed after that. But with Tiger, Mac OS X is finally showing signs of life in this area. Consider my extensive coverage "making up for lost time."

My Mac OS X metadata proposal from 2001 was partly pragmatic, suggesting "emergency fixes" for the short term, and partly an "I have a dream"-style vision statement. Many of the pragmatic suggestions have become less relevant with the passage of time, but the dream is still alive—at least in me. While not wholeheartedly subscribing to my vision, Tiger fulfills a surprising number of my wishes. Here are some excerpts from the proposal followed by the relevant Tiger features.

Create a new volume format that supports arbitrarily extensible metadata and robust data integrity (e.g. journaling), while providing extremely high performance.

As it turns out, Apple didn't have to create an entirely new volume format to add either of these features. Journaling arrived first, and now arbitrarily extensible metadata has arrived in Tiger—all while using the venerable HFS+ volume format.

I still think a brand new file system would offer many more opportunities for increased performance, and I still hope that there's one in development at Apple. But in the meantime, the HFS+ retrofit is certainly better than nothing.

Lead, initiate, or participate in the creation of open standards for file metadata representation. MIME was a start, but it is limited. Robust, hierarchical, extensible, and above all, open standards for file metadata representation are necessary for the future of computing. Apple should work with the rest of the industry on this problem.

Apple has yet to tackle the larger problem of generic file metadata representation and interchange. But as far as file types are concerned, Apple did an excellent job with the creation of Uniform Type Identifiers. They dropped the ball on the "open" part, however.

Add a "metadata services" framework to the OS for converting to and from various metadata representations (e.g. MIME, type/creator, file name extensions).

Apple has provided APIs for converting the lesser file types into UTIs. The framework envisioned in the proposal was more ambitious, however, providing conversion APIs for external (i.e., on disk) as well as internal representations.

Metadata determination strategy summary:

• Try to understand every possible metadata representation.
• Prefer proper Mac OS X metadata over all other representations.
• "Promote" files to proper Mac OS X metadata when possible.
• The outcome of all metadata determinations in Mac OS X should be a
  piece of proper Mac OS X metadata, regardless of which pieces of
  information contributed to the determination.

Tiger does try to understand most file type representations, and UTIs are the recommended internal file type representation. But there's no recommendation of, or implicit mechanism for, the "promotion" of any of the lesser file type representations to UTIs, at least as far as the file's state on disk is concerned.

The final item is one of the most important reversals of historic practice in Mac OS X. Tiger rationalizes the world of file types by providing Mac OS X with a a single, "native" file type that doesn't suck. It will take a while for applications to move away from making decisions based on file name extensions, MIME types, and type and creator codes, but at least now the migration path is clear.

So, after four years of waiting, it seems that at least a few of the features I had hoped for have arrived. As a bonus, Tiger includes a powerful search and indexing service in the form of Spotlight.

Although marred by a sometimes infuriating user interface, and perhaps overly cautious in its decision to largely ignore Tiger's new, native file system metadata features (extended attributes), the core functionality of Spotlight is impressive and immediately useful. The public interfaces are also admirably implementation agnostic, so there is hope (if not outright expectation) that technical improvements will arrive in the future without breaking compatibility with the current API.

I hope at least a few readers enjoy this topic as much as I (clearly) do. For the rest of you, even if you just skimmed up to this point, there are a few key points about metadata in Tiger that I want you to take away.

• Tiger includes support for arbitrarily extensible file system metadata via its extended attribute APIs. Extended attributes are natively supported on HFS+ volumes (no reformat required) and are emulated via "._files" elsewhere. Extended attributes are required to support ACLs, but have the potential to do much, much more in the future.
• Access Control Lists (ACLs) are a much more flexible system for controlling who can do what to which files and directories. This is primarily a server feature, but the kernel support and APIs for ACLs are in both versions of Tiger.
• Spotlight is a search and indexing service. It is only related to file system metadata insofar as any third-party metadata importer plug-ins might reference it. Spotlight does not create or modify metadata; it only reads, indexes, and searches it. There's been a widespread misattribution of file system metadata superpowers to Spotlight in the Mac community. The real hero in that area is the new extended attributes system.
• Uniform Type Identifiers (UTIs) provide a powerful new standard for data typing that surpasses MIME types, type and creator codes, and file name extensions. UTIs are the new standard for internal data type representation in Tiger, but they do not currently have an external (on-disk) incarnation. Future directions in combination with extended attributes look promising, but I am still unsure of Apple's long-term goals in this area.

I'm sure that was a heck of a lot easier to read than the past umpteen pages, but there's more to technology than executive summaries. Nevertheless, it can actually be boiled down even further, perhaps to a single bullet point suitable for the Tiger retail box. "Mac OS X 10.4 Tiger: finally doing something about file metadata." Yay.

Quartz

Quartz is the umbrella term for the Mac OS X display layer through which all drawing and screen display is done. Conceptually, the most important parts of Quartz are:

• Quartz 2D is a vector-based drawing API. The functions it provides map nicely to the drawing commands that make up Adobe's PDF page description language. Quartz takes advantage of this to enable nearly anything drawn to the screen to be written to a PDF file, and to help unify printing and screen display under a single API.
• The Quartz Compositor is the engine that combines ("composites") the rasterized output of each application's drawing commands into a single scene.
• The Window Server manages all the windows on the system, tracking the input focus and routing events (e.g., mouse or keyboard actions) to the appropriate applications.

Practically speaking, the Quartz Compositor exists inside the window server, and Quartz 2D is implemented as a series of libraries. Mac OS X has other drawing APIs in addition to Quartz 2D. OpenGL, QuickDraw, and QuickTime are all ways to produce rasterized content that the Quartz Compositor must then add to the screen.

Since the Quartz Compositor only works with rasterized data (i.e., bitmaps), the result of every drawing command must be stored as a bitmap in a memory buffer somewhere. These memory buffers are also called "backing stores," and they exist for every window on the system.

The Quartz Compositor reads all of these backing stores and combines them into a single scene, which is then moved in the frame buffer of the video card. The frame buffer is essentially the backing store for the single "image" that makes up the entire screen.

Conceptually, the system looks like this:

Things get more interesting when we start to consider which pieces of hardware are involved in which parts of the system. Although the conceptual model has remained the same, the implementation has changed radically as Mac OS X has developed.

OpenGL has been completely hardware accelerated from the start. It is, after all, one of the two APIs that modern video cards are designed to accelerate (DirectX being the other one). We'll discuss QuickTime acceleration in a later section. That leaves Quartz 2D and QuickDraw, the two general purpose drawing APIs in Mac OS X.

Quartz 2D is Mac OS X's "native" drawing API. It was created pretty much from scratch for Mac OS X 10.0, and it remains the recommended way to draw to the screen. QuickDraw is the venerable, revolutionary drawing API from classic Mac OS that enabled the first mass-market GUI operating system to run on an 8MHz Motorola 68000 CPU attached to 128KB of RAM. QuickDraw is supported in Mac OS X in order to provide backward compatibility for Carbon applications ported from classic Mac OS.

Quartz implementation history

When Mac OS X was introduced in 2001, the division of labor looked like this.

Quartz in Mac OS X 10.0 - 10.1

The bandwidth numbers are slightly misleading because they represent the state of the art in 2005: a Power Mac G5 with an ATI X800 video card. No such beast existed when Mac OS X 10.0 was introduced. I chose to keep the bandwidth numbers the same to highlight the architectural differences of the various implementations of Quartz. The hardware differences will be a factor eventually, but in the meantime just consider the ratios between the bandwidth numbers rather than their absolute values.

In the Mac OS X 10.0 Quartz implementation, it's clear that the CPU is doing a lot of work. Every major component of the system executes on the CPU, and the CPU is involved with nearly every transfer of data. The sole exception is the hardware-accelerated final drawing to the screen. (Well, to the frame buffer really. The video card handles flushing the frame buffer to the screen.)

Note that only the Quartz Compositor gets direct, hardware-accelerated access to the frame buffer. Applications can't have direct access to the frame buffer because the Quartz conceptual model requires the output from all the applications to be blended together to create the final scene. Everything has to go through the Quartz Compositor.

Since the Quartz Compositor's primary job is to blend the backing stores for each window into a single, cohesive scene, accelerated access to the frame buffer isn't much good until the actual work of compositing is completed. Then, yes, the finished scene can be squirted into the frame buffer with no CPU involvement.

This level of hardware acceleration is actually a step down from what existed in classic Mac OS. The QuickDraw display model used in classic Mac OS was not a compositing engine. Instead, individual applications "owned" the portions of the screen where their windows were visible, and could flip pixels on and off in those areas without worrying about what other applications were doing. Direct frame buffer access was therefore granted to any application that wanted it.

On top of that, contemporary video cards supported what was called "QuickDraw acceleration." QuickDraw drawing commands were translated by the video driver and then sent directly to the video card. The video card would then do the actual drawing, modifying its frame buffer as appropriate to draw the lines, shapes, or whatever else the commands dictated.

Applications running in Mac OS X 10.0 had none of those advantages, and it showed. Mac OS X 10.0 felt slow. The CPU had to work overtime, first to decide what to draw in response to each applications' drawing commands, and then to composite the backing stores for each window into a final scene. That's a lot of drawing, a lot of blending, and a lot of traffic going between the CPU and main memory (RAM). Also remember that in the days of 10.0, the bus between the CPU and RAM was 100MHz—if you were lucky! It was painful.

The Quartz software was further optimized in Mac OS X 10.1, and the hardware got marginally faster. (Woo! A 133MHz bus between CPU and RAM!) But the division of labor remained the same until Mac OS X 10.2 Jaguar in 2002. Jaguar's implementation of Quartz looked like this.

Quartz in Mac OS X 10.2 - 10.3

Jaguar effectively moved the Quartz Compositor onto the video card. It did this by turning the window server into an OpenGL application. Each window was treated like a 2D surface, with the backing store as its texture map. Since the video card was already designed to accelerate OpenGL, this refactoring made the Quartz Compositor a perfect fit for the abilities of GPU.

This hardware acceleration of the Quartz Compositor was called Quartz Extreme. It only worked on video cards that supported arbitrary texture dimensions (since a window backing store can potentially be any size) and required a 2x AGP bus in order for the video card to efficiently read backing stores directly from RAM.

Moving the Quartz Compositor onto the video card reduced Quartz's CPU load tremendously. All that tedious blending of pixels is exactly what the GPU is good at. It also got the CPU out of the business of ferrying pixels from the backing store into the video card. Pushing pixel data from place to place is definitely not a good use of CPU power, especially when front-side bus speeds are still stuck below 200MHz.

The effects of Quartz Extreme were easy to feel. My favorite test was to shake a transparent window on top of a playing QuickTime movie. In order to comply with the conceptual model of Quartz, the area under transparent window had to be re-composited every time either the window moved or another frame of video was displayed.

Without Quartz Extreme, on what was a fast Mac for the time (a Power Mac G4 800MHz), shaking the transparent window used so much CPU time that the QuickTime movie, starved for cycles, dropped to less than half its full frame rate. Layering five more stationary transparent windows on top of the QuickTime movie while still shaking the top window effectively used 100% of the CPU. The QuickTime movie simply stopped playing entirely: 0fps.

With Quartz Extreme enabled on the same Mac, the QuickTime movie stayed at a rock-solid 24fps the entire time. I actually tried layering on up to 25 windows before I gave up. The frame rate of the QuickTime movie never wavered.

Like Mac OS X 10.1, version 10.3 Panther added software optimizations to Quartz but did not change the division of labor. So, what's next?

Quartz in Tiger

With the advent of Quartz Extreme, the CPU no longer had to blend window buffers or push them to the video card. Unfortunately, window buffers don't spring fully formed from the application into the backing store in RAM. Something actually has to draw the contents of each window into the backing store.

A Safari window, for example, is full of text, graphics, buttons, and all manner of things. Sure, it's great to have the finished, fully drawn Safari window blended onto the screen by Quartz Extreme without any involvement from the CPU. But unless the Safari window is transparent (or buried under one more transparent windows), that task pales in comparison to the task of actually drawing the window contents in the first place.

Just think of the text alone. All those little characters need to be rasterized from their vector font definitions, then sub-pixel antialiased and blended onto the background. This all has to be done by the CPU—even the blending. Remember, Quartz Extreme only accelerates the Quartz Compositor, and the Quartz Compositor only deals with completed window buffers. Any blending that has to be done in order to actually draw one of those window buffers is up to the CPU. Only after all drawing is complete is a window buffer passed off to the Quartz Compositor.

Drawing takes place whenever a window's contents change. Aside from the initial drawing of a window's contents, two other common cases spring to mind: scrolling and resizing windows. Scrolling can cause the entire contents of the window to be redrawn many times a second. Depending on how the contents of a window reflow, resizing can cause the same thing to happen.

Again, Quartz Extreme does nothing to help here. It's just twiddling its virtual thumbs over on the GPU, waiting for the CPU to finish drawing the contents of the window buffer. Then it quickly blends the completed window buffer onto the screen without breaking a sweat and begins waiting for the next iteration of the window's contents to be drawn.

In this situation, drawing speed can be quantified by a "frame rate" (number of images drawn per second) much like in a 3D game. To borrow (and slightly modify the meaning of) some more 3D gaming terminology, in the days before Quartz Extreme, Quartz was often "fill-rate limited." In situations that required large areas of multi-layered blending, the Quartz Compositor simply couldn't compose the scene fast enough to maintain a good frame rate for the entire display.

In the post Quartz Extreme era, drawing is often "CPU bound." Composing the finished scene by blending all the window buffers is no longer the bottleneck. The GPU eats that task for breakfast. Instead, the system spends most of its time waiting for the contents of those window buffers to be produced in the first place.

Clearly, the solution is to dramatically increase drawing speed, but how? To answer that question, take another look at the two earlier Quartz implementation diagrams. In addition to bandwidth and CPU involvement, there's something else to consider about the data flow lines that connect the various parts of the system: the size of the data that must flow down those lines.

The results of the drawing process (rasterized images) are big and require lots of bandwidth. Drawing commands are small. The best solution is to make sure the biggest data travels over the biggest pipes, leaving the small pipes to deal with the smaller data.

The Jaguar/Panther Quartz implementation doesn't look too great in this light.

Quartz in Mac OS X 10.2 - 10.3

The CPU is used to produce rasterized drawing results that are stored in RAM. That process has about 5GB/s of effective bandwidth on a fast Power Mac G5. When the drawing is complete, the video card has to pull the finished image from the backing store in RAM. Although this does not require CPU involvement, the video card can only pull data from RAM at about 2.1GB/s over an 8x AGP bus. This is the lowest bandwidth connection in the diagram, but it is tasked with carrying the largest pieces of data.

In addition to bandwidth constraints, there is significant synchronization overhead inherent in the Jaguar/Panther implementation of Quartz. When an application finishes drawing to a window backing store, it signals to the Quartz Compositor that its window is ready to be drawn ("flushed") to the screen. To flush a window, the Quartz Compositor has to read the window's backing store from RAM (over that narrow pipe) and then blend it into the scene.

While the window buffer is being read by the Quartz Compositor, the application can't write to it. In other words, the next "frame" of window contents cannot be drawn until the Quartz Compositor is done slurping up the last frame. The reverse is also true: the Quartz Compositor cannot begin reading pixels from the window backing store until the application has finished drawing to it.

The upshot is that, with drawing and compositing running on separate hardware (CPU/GPU) with separate local memory (RAM/VRAM), the resources shared between them (window backing stores in RAM) cause a significant synchronization bottleneck in the drawing system. Combined with the narrow path to the shared resource (relative to the large data it is asked to carry), this effectively limits the maximum frame rate of the drawing system in Jaguar and Panther.

The solution is simple: eliminate both the synchronization and bandwidth bottlenecks by moving Quartz 2D into the part of the block diagram with the fat, red lines: the video card. Enter Quartz 2D Extreme (Q2DE).

Quartz 2D Extreme

What Quartz Extreme did for the Quartz Compositor, Quartz 2D Extreme does for the Quartz 2D drawing API. Here's what the Quartz implementation looks like in Tiger.

Quartz in Mac OS X 10.4 Tiger

If you look at the three Quartz implementation diagrams in sequence, you can see how the video card portion of the diagram has slowly expanded over the course of four years to encompass more and more of the display layer. The reason is clear when you look at the bandwidth numbers: 30GB/s between the GPU and VRAM, and that number is climbing rapidly—much more rapidly than the bandwidth between the CPU and RAM.

As you can see, Quartz 2D is now running on the GPU. This is Quartz 2D Extreme. The only thing left for the CPU to do is to send (relatively tiny) drawing commands to the video card through the driver.

It would be pointless to run Quartz 2D on the GPU if it sill had to write its drawing results into RAM and then force the Quartz Compositor to read them back in again. So Quartz 2D Extreme also moves the window backing stores into VRAM.

Now the data flow looks quite nice. The application, running on the CPU, uses the Quartz 2D API to draw the contents of its windows. Tiny drawing commands are sent from the CPU to the video card through the card's driver. Quartz 2D runs on the GPU, producing drawing results which are then written into the backing store in VRAM over a ridiculously fat 30GB/s pipe. When the drawing is done, the Quartz Compositor pulls the finished backing store from VRAM over the same fat pipe, blends it with all the other backing stores to form the final scene, and sends the finished screen image to the frame buffer over another giant pipe.

Synchronization overhead is greatly reduced by the speed with which backing stores can be read and written, and by the fact that both the producer and the consumer of backing stores are now running on the same piece of hardware and have identical, high-bandwidth connections to the same pool of local RAM.

The devils in the details

Nothing's ever quite as rosy as a high-level block diagram makes it seem, and Tiger's Quartz implementation is no exception. Let's start with Quartz 2D Extreme. While the drawing commands may be tiny, there will occasionally be larger chunks of data that have to be pushed up to the video card. Imagine displaying an image, for example. That image has to get into the backing store somehow. The backing store is in VRAM and the image is on disk. Unfortunately, the video card can't pull data directly from the disk into VRAM. So every once in a while, the CPU will have to insert a larger chunk of data (like an image) into the stream of tiny drawing commands that it sends to the video card.

An overload of "big chunks" in the command stream is actually a potentially disastrous condition. Although the path between RAM and the video card is not shown in the Tiger Quartz diagram, you can see it in the Jaguar/Panther diagram. That pipe is the smallest one in the diagram, less than half the speed of the next smallest pipe, the one between the CPU and RAM. Trying to stuff large pieces of data down this pipe will kill any performance advantages Quartz 2D Extreme might offer.

The solution is to send the big chunks (images and other bitmaps) to the video card only once, and then keep them cached in VRAM. Any subsequent drawing commands that reference those bitmaps can then execute immediately, pulling the data from VRAM over a fat pipe.

A huge number of common drawing operations fit this "upload, cache, and reference" mold. For example, nearly every user interface element in Mac OS X is a bitmap: buttons, checkboxes, window widgets, window background textures, etc. The first time these UI elements are drawn, the bitmap graphics are uploaded to the video card and cached in VRAM. All subsequent UI widget drawing commands can then execute as fast as possible, pulling bitmaps from the VRAM cache as needed.

Perhaps surprisingly, text is the other common example. The first time text is drawn at a particular size with a particular font, the characters shapes (glyphs) are read from the vector-based font definition and then rasterized into bitmaps at the specified size. These rasterized glyphs (bitmaps, really) are then uploaded to the video card and cached in VRAM. All subsequent text drawing using the same font and size can then simply issue small drawing commands ("draw a capital letter 'A'") without any further need to upload bitmaps. Since most text consists of relatively long sequences of glyphs in a few fonts and sizes, this is a big win in practice. Of course, ransom notes that use a different font and size for every single character might not benefit as much...but then, that really depends on how much VRAM you have, doesn't it?

This leads us to the most painful reality of the brave new world of GPU-powered drawing: VRAM is finite. The simple block diagrams shown earlier sweep this reality under the carpet. What happens when there's simply not enough room in VRAM to cache all of the backing stores and bitmaps and rasterized glyphs and who knows what else? Does performance fall off a cliff?

As it turns out, VRAM has been "virtualized" by Mac OS X since Quartz Extreme debuted in Jaguar. Although the Jaguar Quartz diagram shows the backing store in RAM, the Quartz Compositor is smart enough to cache those backing stores in VRAM as well. The biggest limitation of Jaguar's Quartz implementation is that the actual drawing is still done into the backing store in RAM, so the diagram accurately reflects the sequence of events during an actual drawing operation. But as long as a window's contents don't change, the Quartz Compositor can continue to use its VRAM cache of the backing store instead of reading it from RAM every single time.

Implementing even this limited form of VRAM caching required facing up to the reality that VRAM won't always be able to hold cached copies of all of the backing stores. Worse, the amount of VRAM varies depending on the video card being used. To simplify the Quartz implementation, Jaguar needed some way to make VRAM look "limitless" even though it clearly isn't.

This problem has been solved before. The virtual memory system in a modern OS makes RAM look "limitless." Well, okay, it makes it appears as if it is 2³² or 2⁶⁴ bytes long, for 32-bit and 64-bit CPUs, respectively. But that's almost certainly larger than the amount of physical RAM installed (particularly in the 64-bit case).

Although the details are different, this is essentially what Jaguar did with VRAM. To the operating system, VRAM looks a lot larger than it actually is. Quartz handles the details of swapping data in and out of VRAM as needed, using a replacement algorithm tuned to keep the most frequently used pieces of data in VRAM as much as possible.

In Jaguar's Quartz implementation, any backing stores cached in VRAM are simply redundant copies of the backing stores in RAM. All backing stores must exist in RAM in Jaguar because that's where drawing actually takes place. Quartz drawing commands cause the backing stores in RAM to be modified. The completed backing store is then (DMA) transferred to the video card where the Quartz Compositor blends it into the scene and (perhaps) caches it in VRAM, just in case it needs to be used again at some point before it's modified (in RAM, remember) by the application and needs to be re-imported into VRAM.

In Tiger with Quartz 2D Extreme, Quartz 2D drawing commands now modify the backing store in VRAM. The Quartz Compositor, also running on the video card, reads from the very same backing store in VRAM. The backing store in RAM is no longer needed at all.

Well, theoretically, anyway. Again remember that VRAM is finite. What doesn't fit in VRAM has to be stored in RAM instead. Once VRAM is full, there is a constant dance of data moving between RAM and VRAM as needed to (ideally) keep the most frequently used data in VRAM. There's also another important reason a backing store might be in RAM instead of VRAM.

With the introduction of Tiger, Apple has announced that QuickDraw is now deprecated. There will be no more improvements to QuickDraw, and it will disappear from Mac OS X "someday." (That's about as specific as Apple will get at this point.) Apple wants everyone to move to Quartz 2D; this has been the consistent message since Mac OS X 10.0. Developers have had over four years get on the Quartz 2D train. Deprecating QuickDraw is the final shout of "all aboard!"

In Tiger, Apple provides one more big reason for developers to move away from QuickDraw. If an application uses QuickDraw to draw anything in a particular window, that window is entirely exempt from Quartz 2D Extreme acceleration. Instead, everything falls back to the Jaguar/Panther model: the CPU does the drawing and writes the results into a backing store in RAM.

This may seem like a drastic decision on Apple's part, but it makes sense. QuickDraw has over twenty years of undocumented behaviors that would have to be preserved perfectly in any new GPU-powered implementation. QuickDraw's drawing model is also very different from that of Quartz 2D. Getting the two APIs to coexist would not be easy. Put bluntly, deprecated APIs don't get radical changes like this. Apple has said "no future QuickDraw development," and they mean it.

Unfortunately, even Quartz 2D isn't entirely accelerated under Quartz 2D Extreme. In cases where the current crop of video cards cannot perform a particular drawing operation, or cannot do so with a quality that matches the software implementation, those operations continue to run in software mode in Tiger. Thankfully, these operations are uncommon (complex stroked paths, lines with mitered joints and end-caps, etc.) As video cards improve, I imagine even these operations will be supported eventually.

That brings up perhaps the biggest caveat about Quartz 2D Extreme. Like Quartz Extreme before it, Quartz 2D Extreme is not supported by all video cards. It requires an ATI Radeon 9600 or NVIDIA GeForce FX or better. In terms of technology, Quartz 2D Extreme requires support for the ARB_fragment_program OpenGL extension.

Yes, that's right, Quartz 2D Extreme is like Quartz Extreme in another way: it too uses OpenGL to do its work. All those "drawing commands" in the Tiger Quartz diagram are actually OpenGL commands. It is becoming more and more accurate to describe the entire screen in Mac OS X as one giant OpenGL scene.

Tiger's Quartz performance

Given all of the caveats and gotchas described above, what kind of performance boost does Quartz 2D Extreme give? Comparing primitive operations puts Quartz 2D Extreme in the best possible light, revealing the theoretical maximum performance. This table shows the speed of several Quartz 2D drawing operations running in both software (without Q2DE) and hardware (with Q2DE).

Quartz 2D (Software) vs. Quartz 2D Extreme (Hardware)

Some of the results may look surprising, but most make sense if you think about them in terms of the earlier implementation diagrams. Consider the huge speed-up (236x) when drawing an 800x800 filled rectangle. That's basically a fill-rate test: how fast can the drawing API put 640,000 pixels into the backing store and then get that backing store through the Quartz Compositor and into the final scene in the frame buffer.

Quartz 2D running in software using the CPU has to write to the backing store in RAM, then that backing store has to be moved to the video card where the Quartz Compositor can put it into the final scene. Quartz 2D Extreme running on the GPU writes those 640,000 pixels into the backing store in VRAM, which is exactly where the Quartz Compositor needs to read it from. No costly transfer of the backing store from RAM to the VRAM over a narrow pipe is required.

The fill-rate prowess of Quartz 2D Extreme explains the other "area drawing" results: the 128x128 image test shows more of a speed-up than the 8x8 pattern test, and so on.

The 12 character string test is closer than you might expect, given the glyph caching system described earlier. The "mere" 2.7x speed-up for Quartz 2D Extreme in this test reveals just how fast the software implementation of Quartz 2D has become in Tiger.

It turns out that the software-mode optimizations in Tiger are nearly as dramatic as the hardware acceleration features. Take line drawing as an example. Quartz 2D was notoriously slow at drawing lines in Panther and earlier versions of Mac OS X, so much so that developers often chose to use (or in most cases, "stick with") QuickDraw for such tasks.

Needless to say, it doesn't look good when your deprecated drawing API is faster than your shiny new one. Something had to be done in Tiger, and boy was it ever.

It's hard to see past the giant bar for hardware-accelerated Quartz 2D Extreme, but look again at the software-mode Quartz 2D performance. Quartz 2D running in software more went from about half the speed of QuickDraw in Panther to being over five times as fast as QuickDraw in Tiger.

Text drawing has seen similarly dramatic speed increases over the life of Mac OS X. In Tiger, hardware accelerated text drawing via Quartz 2D Extreme is "only" about three times as fast as software mode. We'll revisit the topic of software optimization vs. hardware acceleration in a later section. For now, suffice it to say that software-mode drawing has made great strides in Tiger.

All of these synthetic benchmarks are interesting, but how does this new technology fare in real-world tests? Obviously, few applications will exhibit the performance boosts seen in the most impressive synthetic benchmarks, but it's not as cut and dried as you might think. Some applications are actually a bit slower when running under Quartz 2D Extreme.

What explains this? It all comes back to VRAM virtualization. Quartz tries to be smart about what it keeps in in VRAM, but it can't possibly predict the future behavior of applications without some help. When running under Quartz 2D Extreme, applications are expected to retain references to the CoreGraphics data structures that they want cached in VRAM. (CoreGraphics, or CG, is the framework that implements most of Quartz 2D.) The VRAM virtualization system uses this as a hint that those pieces of data are likely to be used again in the near future. (It's just a hint, mind you. Retaining the reference does not guarantee the referenced item a permanent place in VRAM.)

Unfortunately, this has not been a common practice in Mac OS X applications. Prior to Quartz 2D Extreme, keeping references to large pieces of data around long after the things they refer to have been used was usually considered a waste of memory for no significant performance gain. Remember, in Panther and earlier, drawing with a large number of pixels was likely to be limited by the upload bandwidth from RAM to VRAM more than any other factor. If a bitmap was used for repeated drawing, Mac OS X's excellent disk caching system made it less critical to retain the bitmap in the application's memory. It made just as much sense to discard the reference and then get the bitmap from memory anyway (the disk cache) the next time it was needed.

When running under Quartz 2D Extreme, the rules change. VRAM is the place to be, and anything that's not there has to get there before drawing can begin. Leaving all the big pieces of data in RAM instead of VRAM is like combining the overhead of the Panther and Tiger approaches. This is why an application that discards all of its graphics data references as soon as it's done with them may actually get slightly slower when running under Quartz 2D Extreme.

It's an easy problem for a developer to fix, however, and they should be used to it by now. Every time a major new version of Mac OS X is released, it behooves developers to profile their applications again. As Apple has warned on many occasions, "What was once cheap may now be expensive and vice versa. Don't assume; measure!" The price of performance is eternal vigilance.

I suspect developers of graphics-intensive applications will be highly motivated to do this for Tiger. All it takes is one look at the performance numbers for Quartz 2D Extreme to see that the the potential benefit of retaining a few more CG references is well worth the development cost.

As for developers who are still using QuickDraw, well, they've had four years. They probably have another two at least before QuickDraw disappears completely, but honestly, at some point it's time to blit or get off the pot.

There's one final barrier to hardware-accelerated bliss. Quartz 2D Extreme is disabled by default in Mac OS X 10.4.0. That's right, the whiz-bang new technology you just read all about is not actually used in Tiger unless it's explicitly enabled using the Quartz Debug application. Even then, it only applies to applications that are launched after it was turned on. It also appears that Q2DE is re-disabled when you quit the Quartz Debug application.

Why develop something as impressive as Quartz 2D Extreme and then leave it turned off by default? My inquiries to Apple have gone unanswered, so I can only speculate about the reasoning behind this decision. My best guess is that all of the bugs could not be excised from Q2DE in time for Tiger's launch date, and that it will be enabled by default in a subsequent update—perhaps as early as version 10.4.1.

Quartz summary

The Quartz display layer has made radical improvements in hardware acceleration in every even-numbered release (Quartz Extreme in 10.2 and now Quartz 2D Extreme in 10.4), with software-only optimizations in the odd-numbered releases. Tiger is unique in that it provides tremendous software optimization in addition to a new level of hardware-acceleration.

Quartz 2D Extreme is impressive already, but it has the potential to become even faster and more powerful. (The first step is for it to be enabled by default, of course.) Video cards are evolving much more rapidly than the other major hardware components of the modern PC (CPU, RAM, disks) in terms of both performance and capabilities. Getting (nearly) all drawing functionality off the CPU and onto the GPU sets up Mac OS X nicely to reap the benefits of future video card innovations. As we'll see in the next section, Quartz 2D isn't the only API that wants to get in on the GPU gravy train. The video card is getting pretty crowded these days...

Core Image

Core Image is Tiger's new image processing framework. It joins Core Audio, Apple's existing audio processing framework. The technologies share more than just a naming convention. They both provide a plug-in architecture for processing their respective media types. These plug-ins are called "units." Core Audio uses Audio Units and Core Image uses Image Units.

Core Image is conceptually very simple. An input image is passed through a series of chained Image Units, resulting in an output image.

Core Image

The entire image pipeline uses 32-bit floating-point values for each of the four channels (red, green, blue, and alpha) in order to preserve as much accuracy as possible across many calculations. Tiger ships with 60 predefined Image Units covering most common image manipulation tasks: blurs, distortions, color adjustments, etc.

Image processing doesn't get much more straight-forward than the diagram above. But it's in the implementation where Core Image really shines. Two aspects of Core Image make it a lot more attractive than the diagram above might suggest.

Although Core Image is conceptually a chain, with the output one one Image Unit becoming the input for the next, a naive implementation of this design would be prohibitively slow. The performance would degrade roughly linearly as the number of Image Units increased, and there would be a tremendous amount of i/o associated with reading and writing all the intermediate results.

A naive implementation of Core Image

This is not how Core Image is actually implemented in Tiger. Instead of running through each Image Unit in sequence, producing an intermediate image after each step, Core Image uses what's called "lazy evaluation."

Passing an image through an Image Unit doesn't actually cause anything to be rendered. The output of an Image Unit is simply a "recipe" for how to produce the desired result. As the image passes through all of the Image Units, each Image Units appends more steps to the recipe.

Actual rendering is only done when absolutely necessary. Usually, that's at the end of the sequence of Image Units. At that point, the entire recipe is actually run, producing the final result image. So the implementation looks a lot more like this.

Core Image in Tiger

There's just one "do work" stage, no (potentially large) intermediate results to read and write, and a single "effective Image Unit" (the big cloud in the diagram) that's combination of all of the actual Image Units.

The second clever aspect of Tiger's Core Image implementation is how it produces and runs the code that makes up the "effective Image Unit." Core Image includes a just-in-time (JIT) compiler that converts the implementation-agnostic "recipe" for the entire Image Unit chain into optimized code, targeting the fastest hardware available at the time of execution.

In many cases, that will mean AltiVec. On multi-processor Macs, the Core Image JIT compiler will even create SMP-aware code that utilizes all available processors. Since this code is created on-the-fly and is tailored to the specific set of Image Units that contribute to the overall recipe, the potential for optimization is limited only by the intelligence of the JIT compiler.

If a reasonably modern video card is installed, AltiVec may not be the "fastest hardware available." Instead, the Core Image JIT compiler may target the GPU, producing (essentially) a pixel shader to run the recipe via OpenGL. This is right in the GPU's sweet spot, so the job gets done fast.

How fast? The standard Core Image demo shows a handful of Image Units being applied to a source image, each one executing instantly as it is applied. Once the entire recipe is complete, any one of the Image Units can be adjusted—increasing the radius of a blur or moving the center of a distortion effect—and the image will update in real time. It's impressive.

Even more impressive is this: adding Image Units seems to have no affect on performance. Well, at least initially. So long as the Core Image JIT compiler produces code that can execute comfortably on the current GPU, there will be no discernible performance difference between a chain of two Image Units and a chain of eight. But once the recipe requires more than the number of rendering passes that the GPU can execute in, say, 1/30th of a second, performance starts to slow perceptibly.

Anyone who sees a Core Image demo can't help but think of Photoshop. Core Image seems to do the same thing Photoshop does with its filters, only Core Image does it in real time. Visions of a "Photoshop killer" powered by Core Image quickly spring to mind.

Unfortunately, this vision suffers from a very narrow view of what Photoshop actually does. Although Adobe Photoshop was initially (and understandably) identified with its image filtering features when it was introduced in 1990, they are perhaps the least interesting features of the application today.

It's not very difficult for a wannabe "Photoshop killer" application to re-implement any one of Photoshop's filters. (If that was all it took to be a Photoshop killer, the (free) GIMP would be the dominant graphics application today.) The hard part is matching Photoshop's fifteen years of refinement in the areas of drawing and editing tools, file format compatibility and conversion, workflow integration, and all the other things that keep Photoshop customers coming back version after version.

That's not to say that creating a Photoshop killer is impossible, or that it can't or won't incorporate a technology like Core Image. But Core Image doesn't really help the effort much. In fact, as a Mac-only technology, any real Photoshop killer would probably be forced to forego Core Image entirely.

Delusions of grandeur aside, Core Image really is powerful and I'm sure there are plenty of non-Photoshop-killer applications that will benefit from using it. In fact, Tiger actually uses it itself to implement this next technology.

Video in Tiger

Core Video

I left out one of the "Core" technologies when I introduced Core Image earlier: Core Video. That brings the total number of "Core" media technologies to three. As part of the Tiger branding effort, they all have a snazzy new logos that look like glass spheres with gem-like cores.

Core Image

Core Audio

Core Video

I included these icons (and at such a large size) because I think they look cool. Making logos for abstract technologies like Core Image, Audio, and Video is not easy. Whoever created these deserves a raise. They look good enough to eat.

Core Video is a bit of a "secondary" technology, or perhaps a "branding-only" technology. It's essentially the application of Core Image filters to video instead of still images.

That's not to say it isn't impressive. If anything, it highlights the real-time performance of the technology behind Core Image. If you've ever wanted to apply blurs, filters, edge detection, and distortions to live video—and have all of the effects editable in real time—Core Video is the technology for you.

Since Core Video is useless if it does not execute in real time, it requires a programmable GPU. Core Video leverages the GPU via OpenGL, just like Core Image and Quartz Extreme and Quartz 2D Extreme...noticing a trend here?

Applying a series of filters to a still image is one thing. Applying them to video, a series of still images produced on the fly by an entirely separate part of the system, is something else entirely. Not only must Core Video finish processing each frame of video before the next frame needs to be displayed, it must get the frame in a timely manner in the first place to even have a fighting chance.

While this is possible even in earlier versions of Mac OS X, as evidenced by the impressive OpenGL movie player application created by Ars forum member "feelgood," it is much, much easier to do in Tiger using Core Video. That's largely thanks to some significant changes made to the source of most frames of video in Mac OS X: QuickTime.

QuickTime 7

QuickTime 7 is a long-overdue modernization of the oldest continuously compatible digital media system in the industry. Let's start with the changes that make Core Video possible.

It would not be entirely inaccurate to describe the QuickTime-related sessions at Apple's Worldwide Developer Conference over the past four years as a series of decreasingly kludgy examples of how to get QuickTime video to play on a 3D surface.

(Really grizzled veterans of the Apple API wars may recall seeing a purely software-powered demo of QuickTime video playing on all six surfaces of a cube back in the days of "Escher", a.k.a. QuickDraw 3D. Ah, memories...)

In Mac OS X, the "video on a 3D surface" problem has been frustrated by the difficulty of getting each frame of video out of main memory (where QuickTime draws it), converting it into a format that OpenGL will accept as a "texture," and then uploading it to the video card where OpenGL can finally map it onto a surface. Try doing this 30 times each second and you get an idea of the challenge.

QuickTime 7 eliminates the need for kludges entirely. It represents the final unification of QuickTime and OpenGL. Since every other technology that displays or manipulates images is already running on the GPU in Tiger, it only makes sense for QuickTime to move there too. In fact, this change is essential if any other part of the display system wants fast access to QuickTime video frames.

With QuickTime 7 in Tiger, every QuickTime movie is being displayed on an OpenGL surface. It just so happens that the surface is perfectly parallel to the display.

But there's more to QuickTime 7's modernization than just moving the video display onto the GPU via OpenGL. A lot of old, crufty parts of QuickTime's implementation have been superseded by more modern equivalents. Here's what QuickTime looked like in classic Mac OS circa 1995.

Back then, QuickTime had its own audio and video processing engines. It used QuickDraw to draw to the screen and the Sound Manager to output audio, both of which interfaced directly with the hardware.

QuickTime 7 in Mac OS X 10.4 Tiger looks like this.

With QuickDraw deprecated, QuickTime 7 uses Core Graphics (another name for Quartz) to draw to the screen. Core Video (and by extension, Core Image) is used to process video. Core Audio finally replaces the horribly outdated Sound Manager. (Though not shown in the diagram, QuickTime 7 still uses plug-ins to support different codecs, just as it always has.)

Core Graphics, Core Image, and Core Video are all built on OpenGL, which in turn interfaces with the video hardware. Core Audio (which I regret that I have not covered in the past, and will not have time to cover here) is built on its own hardware abstraction layer (HAL), with the actual audio hardware below that.

Overall, the QuickTime 7 architecture is more abstracted, more powerful, and more modern. This new architecture made possible another change to QuickTime that was long overdue: the decoupling of video decoding and video display.

When QuickTime 1.0 was introduced in 1991, its video decompression pipeline was based on a simple, and seemingly safe assumption: frames of video would be decompressed in the order that they were to be displayed. This assumption was built into the design and APIs of QuickTime from the very start, and it continued all the way through to QuickTime 6 in 2004. This meant that a compressed frame of video could only depend on earlier frames of video for its successful decompression.

This turns out to be a problem. Modern video codecs (Windows Media 9, Real Video 10, even DivX and friends) often use knowledge of future frames of video to compress a given frame. QuickTime, being a strict "in order" video decompression engine, has absolutely no knowledge of future frames of video when it's decompressing a frame.

QuickTime 7 eliminates this limitation. Finally, the decode order and display order of video frames can be distinct. The most impressive thing about this change is that the full suite of QuickTime editing abilities remain intact. Think about the difficulty of enabling arbitrary cut, copy and paste when a frame of video may depend on future frames for its decompression. What if you cut a portion of a video clip whose final frame depends on a future frame that is beyond the bounds of the cut operation? How can that video be pasted elsewhere safely?

A solution to this problem is "non-trivial," as they say, and it's a big part of why this architecture change has been so long in coming. QuickTime 7 delivers it. To celebrate, Apple includes a modern video codec of their own in QuickTime 7: H.264.

H.264 is the tremendously unglamorous name for the latest video codec in the MPEG-4 standard. It's also known as AVC (Advanced Video Coding) and MPEG-4 Part 10, both of which make H.264 look good in comparison. It's a thoroughly modern codec that rivals the best the competitors (commercial or open source) have to offer. As a part of the widely recognized and supported MPEG-4 standard, it has a leg up on the single-vendor proprietary competitors (WM9, Real) and more corporate weight behind it than most other third-party or open source initiatives (XviD, DivX, 3ivx, etc.)

A rigorous quality and size comparison of these codecs deserves its own article, so I'm not going to go into it now. Suffice it to say that it's comforting to see QuickTime playing in the big leagues again when it comes to video codecs.

QuickTime 7 also includes improved support for multi-track audio, high-resolution audio, media metadata access, and now allows plug-ins to be written using Objective-C.

Speaking of Objective-C, Tiger includes a QuickTime-flavored gift for Cocoa developers. It has historically been, if not exactly difficult, then at least very annoying to use QuickTime from within a Cocoa application. The two technologies came from very different worlds. QuickTime was designed for classic Mac OS (System 6!) circa 1991, while Cocoa is an evolution of NeXT technology.

Tiger finally bridges the two worlds with QTKit. QTKit provides a complete object-oriented interface to all of QuickTime, with Cocoa-style classes for everything: tracks, movies, views, the works.

When Carbon was first being developed, Apple chose to base one of its "important" applications on the new API to prove its usefulness: the Finder. In the software industry, this is poetically called "eating your own dog food." Any long-time Mac OS X user knows how things turned out in the case of the Finder, but the dog food concept itself is a sound one.

In Tiger, Apple has re-implemented the QuickTime Player application using QTKit, and I have to say, the results are immediately impressive. Granted, many of the most impressive features of the Tiger QuickTime Player are a result of the improvements in QuickTime 7 itself; the Player application is simply providing access to them. But at the very least, it's a validation of QTKit's utility.

Here's a brief glimpse of the new QuickTime Player.

The new Cocoa QuickTime player based on QTKit

The player window keeps the same design, but looks subtly different with yet another unique shade of brushed metal. Player windows now resize opaquely, even while the video is playing, though some frames of video are dropped on slower systems.

The "genie effect on playing a movie" animation moves yet another step closer to perfection in Tiger, with the only discernible flaw being the freeze-frame chosen for the initial Dock icon. (Shift-click on the minimize (yellow) widget of a QuickTime Player window while the movie is playing to judge for yourself.)

There's also a new audio/video control panel.

QuickTime Player's audio/video controls

All of the video adjustments work in real time while the movie is playing. Thanks to Core Video, even slow systems don't break a sweat—provided they have a capable GPU, of course. The jog slider also works in real time, scrubbing through the movie at an adjustable speed in either direction. The slider is not as nice as an actual, physical jog wheel, but it is responsive. I only wish the slider was a bit longer to take some pressure off my precision mousing skills.

My favorite new feature is the playback speed adjustment, particularly how it can increase the speed of the audio without changing the pitch. This means I can watch, say, WWDC sessions at 1.5x speed without making every presenter sound like one of the chipmunks. (There is no filter for Bertrand's accent, unfortunately.)

Obnoxious

There's one final unpleasantry to deal with: QuickTime Pro. Like its predecessors, the Tiger QuickTime Player is crippleware. Many of the most useful features are disabled, and can only be enabled by paying Apple $30 for a QuickTime Pro license. The menu items for these "pro-only" features are grayed out and have obnoxious little "PRO" badges next to them.

Don't misunderstand, none of the disabled features in the QuickTime Player are actually disabled in QuickTime 7 itself. They're simply disabled in the player application. Anyone can create a QuickTime player application that goes right ahead and calls the APIs that the QuickTime Player in "non-pro" mode refuses to call. Oh and by the way, the QuickTime browser plugin-suffers from the same intentional feature debilitation.

This has been the case for years with QuickTime, so why get all riled up about it now? Here's why. Mac OS X ships with a complete integrated development environment that supports C, C++, Objective-C, Java, and all of the APIs in Mac OS X (not to mention distributed compiling, a GUI design and layout tool, and a suite of performance monitoring applications). Tiger includes a free web browser, e-mail client, address book, dictionary, thesaurus, font manager, and AIM/Jabber instant message client. When you buy an iMac you get all of the above plus iLife: iPhoto, iMovie, Garage Band, and iDVD.

The total development cost of this software bundle is absolutely huge. The total retail cost of iLife alone is $80. And yet after spending $1,500 or more on a new Mac with this great software bundle, what's waiting for you when you fire it up for the first time and try to watch a QuickTime movie trailer in full-screen mode? Why, it's a nag screen asking you to pay $30 more for the "privilege" of calling the QuickTime APIs that are sitting right there in the library code on your disk.

This is just criminally stupid. It mars the otherwise exemplary out-of-box experience for buyers of consumer Macs especially. Having spent well over $4,000 on my current crop of Mac hardware (plus $80 for iLife '05 plus who knows how much for the Mac OS X Public Beta through Tiger), I find it personally insulting that I'm still not entitled to the "wonders" of QuickTime Pro.

Yeah, sure, I can download a third-party movie player application and find a third-party QuickTime browser plug-in. I can watch movie trailers in iTunes, which will go full-screen even without the magic "pro" key. Or I can google for an illegitimate QuickTime Pro key code. I can even shell out the $30. But it's not the money that bothers me, it's the principle. I'd be happy if Apple simply raised the price of its hardware by $30. On a $4,000+ bill, it's practically a rounding error.

But please, Apple, give up on the QuickTime Pro thing. It's always been annoying, but when viewed alongside today's suite of bundled Apple software, it's downright ridiculous. Worse, it makes the Mac platform look bad when the bundled QuickTime Player application can't do all of the things that make QuickTime so cool: cut, copy, and paste together different kinds of media into a single file, extract and recombine tracks, import and export a huge number of formats, and yes, view video in full-screen mode.

Yeesh. Rant over.

QuickTime 7 summary

Despite the ongoing annoyance of the "QuickTime Pro tax," QuickTime 7 is the most important upgrade to QuickTime in the Mac OS X era. It solves long-standing architectural problems, leverages several of Tiger's other new technologies to do things only dreamt of by QuickTime 6 and earlier, includes its own best-of-breed video codec, and is finally embraced by Cocoa. The new QuickTime Player is good enough to be in danger of reinforcing the (largely uninformed) folk wisdom in the Mac community that rewriting an application in Cocoa automatically makes it better. QuickTime 7 has been a long time in coming, but it has turned out to be well worth the wait.

(And if Apple feels like there's nothing left to do for QuickTime 8 except produce plug-ins for the alphabet soup of audio and video codecs, subtitle tracks, and container formats used by those inscrutable Anime fansubbers, you won't hear me complaining...)

Dashboard

The setup

In February of 2003, Arlo Rose and Perry Clarke released a new Mac OS X application called Konfabulator. Konfabulator is an engine for running small programs called "widgets." A widget is a bundle that contains an XML description file, images and other resources used in the user interface, and JavaScript code to glue it all together.

Although Konfabulator ships with several useful widgets, the important part of the product is the Konfabulator application itself. It's the "widget engine" that reads the bundles, parses the XML files, loads the resources, builds and displays the widgets' user interfaces, and runs the JavaScript code to make the widgets actually work.

When running, widgets are self-contained windows that do not have a menu bar of their own. They can be interleaved with other windows, forced to float on top, or pushed back to the level of the desktop background.

Like Kaleidoscope, the popular user interface theme engine for classic Mac OS which was also coauthored by Arlo Rose, Konfabulator is meant to be an environment for hosting user-created content. There is a large and growing collection of user-contributed Konfabulator widgets.

Mac OS X 10.4 Tiger includes a technology called Dashboard which is very similar to Konfabulator. It too is an engine for running small programs called "widgets." Dashboard widgets are also bundles containing a description file, images and other resources, and JavaScript code to make the widgets actually work. The implementation details start to diverge from there, however.

Instead of purpose-build XML files and a custom JavaScript API, Dashboard uses HTML, CSS, DOM, and all the other technologies used in a modern web browser. Dashboard actually uses Web Kit, the engine that powers the Safari web browser, to run its widgets. The widget description files are actually HTML pages, and the JavaScript code behaves just as it would if it was running inside a web browser. In fact, the Tiger version of Safari (2.0) can also be used to display and run widgets.

Dashboard goes a few steps further. Dashboard widgets can be augmented with compiled Objective-C code. This allows them to interact with parts of the system that are not accessible from within a web browser (e.g., Mac OS X's built-in address book database). Apple has also added a few new features to Web Kit in support of Dashboard widgets. There are two new controls (a slider and a rounded search field) plus a JavaScript interface to a subset of the Core Graphics API.

The new Web Kit features are also present in Safari, of course, and this has caused much controversy in the web developer community. I side with the other web developers here. I wish the new, proprietary Web Kit features were confined to Dashboard where they are appropriate and useful. I see little value, and much danger, in allowing them to work in Safari by default.

I pessimistically predict a proliferation of slider controls and rounded search fields on Mac-centric websites in the wake of Tiger's release. Please, Mac web masters, exercise restraint. Don't make this the MARQUEE tag all over again.

Although Tiger ships with several useful widgets, Dashboard is designed to support user-created content. Apple has already run several Dashboard widget-building contests, and third party widget collections began to pop up months before Tiger was released.

Finally, rather than being mixed with other windows on the system like Konfabulator widgets, Dashboard widgets exist in their own private window layer. The Dashboard layer is not visible by default. When Dashboard is activated, the screen dims and the active Dashboard widgets swoop in from all sides of the screen.

Dashboard can be activated by typing a key-combination, dragging the mouse to a corner of the screen, or clicking the Dashboard Dock icon. (The key-combination and screen corner are user-configurable.) Clicking anywhere outside a widget or re-triggering the Dashboard activator causes all the widgets to swoop back out of view. You can see a movie of Dashboard in action at Apple's web site.

The controversy

There has been a lot of debate surrounding Dashboard. Many members of the Mac community see it as another example of Apple hurting its software developers by creating a product that does roughly the same thing as an existing third-party product, and then bundling it for free with the operating system.

There is no denying that Dashboard hurts the commercial viability of Konfabulator on the Mac platform. The question is, is this good, bad, or indifferent? Is big, bad Apple stomping on an innovative third-party developer, essentially eating its own young and hurting the Mac software market and the whole platform as a consequence? Or should the Konfabulator folks simply accept this turn of events as part of the software development business?

This debate mostly ran its course in the months before Tiger's release. But like many community-wide debates, it produced its fair share of "commonly accepted wisdom" on both sides of the debate, much of which is wrong. Here are the two biggest misconceptions.

"Apple stole the idea of Dashboard from Konfabulator."

This old saw implies that Arlo and Perry somehow "own" the concept of a runtime engine for markup-based widgets with JavaScript glue, and therefore have some sort of automatic legal recourse. They don't. Arlo and Perry do not own any patents on Konfabulator's implementation and do not have exclusive rights to the ideas behind it.

"Konfabulator is actually just a rip-off of Apple's Desk Accessories, a technology released with the original Macintosh in 1984."

This misconception stems from the fact that many widgets perform functions similar to those of some well-known Desk Accessories. There was a calculator Desk Accessory and there is a calculator Dashboard widget. (I don't recall seeing a calculator Konfabulator widget, but one probably exists somewhere.) Of course, there's also a calculator application bundled with Mac OS X.

How about the clock? Alarm Clock was an early Desk Accessory. Dashboard and Konfabulator both have clock widgets. But there's also a clock application that's part of the Date and Time preference panel (of all things) in Mac OS X.

Clearly the idea of an "accessory" application is very old, and has long since spread beyond the bounds of any one application technology—if it was ever really so confined to begin with. Dashboard and Konfabulator owe nothing more to Desk Accessories than the Calculator and Stickies applications do.

Well, how much is that? It's not the quantity, but the significance of the intellectual debt that really matters. At a certain point, ideas simply enter into the public consciousness. The concept of little "accessory" applications has long since passed that threshold. In the same way that new word processors are not seen as "WordStar rip-offs," Konfabulator's historical debt to Desk Accessories is not a significant part of the product's identity.

When Konfabulator was introduced, it was (rightfully) seen as a new kind of thing on the Mac platform. This identity was based on Konfabulator's "big idea," its "primary innovation," if you will. Here's a quote from Arlo Rose's journal.

My idea for Konfabulator was born from wanting to have a really simple run-time environment for people to develop small specialized applications that could look and behave however they wanted. The key point being that it is up to the user to make the cool Widgets. The user would know what they needed, and the user could then create that. [...] It was about empowering the user to make their own little apps that did what they wanted.

This couldn't be more different from the "big idea" that spawned Desk accessories. Arlo agrees.

The [Konfabulator] concept had nothing in the slightest to do with Desk Accessories

Desk Accessories were a technique for launching and running multiple applications on a single-tasking operating system (classic Mac OS). The importance of this idea has disappeared in the modern age of multitasking operating systems. It began to disappear as far back as System 7, when the Mac operating system finally became multitasking by default. Things that were once Desk Accessories shipped as weird little "mini-applications" in System 7. The place where Desk Accessories were traditionally launched from (the always-available Apple menu) could contain any kind of item in System 7, and Desk Accessories could be launched from anywhere that a regular applications could be launched from. In Mac OS X, Desk Accessories disappeared completely, replaced with plain old application (e.g., the Calculator and Stickies applications).

While Konfabulator's primary innovation has nothing to do with Desk Accessories, the "big idea" behind both Konfabulator and Dashboard is the same: to provide a runtime environment that enables the creation of simple applications using widely-known "Internet age" technologies: XML, HTML, JavaScript, etc. No compiling is necessary, and widgets have a completely open format that encourages users to learn from the work of others.

Again, this doesn't necessarily mean that either Konfabulator or Dashboard was the very first product to be based on this idea. What's important is that Dashboard and Konfabulator are both barking up the same tree, and that the Desk Accessories ancestry is about as significant to either of them as it is to the Stickies application.

The worst part of the misconception that Konfabulator and Dashboard are both obvious derivatives of Desk Accessories is that it adds nothing to the debate, regardless of its validity. The not-so-hidden agenda of the Desk Accessories connection is that "Apple did it first," therefore Dashboard did not copy Konfabulator, therefore Arlo and Perry have no reason to gripe.

(Incidentally, posing the question of who copied whom already has the built-in (and correct) assumption that Konfabulator and Dashboard are basically the same thing.)

To assess Arlo and Perry's "right to gripe," a different question has to be asked: Was there a reasonable expectation that Apple would bundle a Konfabulator-like product for free with its next major operating system release? If you (incorrectly) believe that the "big idea" behind Konfabulator originated with Apple's Desk Accessories rather than with Arlo and Perry, it might seem more likely that Apple would build a product based on "its own" idea.

In reality, Apple's likelihood of building an OS feature has little to do with the source of the idea upon which it is based, whether it's an existing third-party product like Konfabulator or an already well-known application genre. It has everything to do with the value the idea might add to the OS.

Of all the possible product ideas floating around the software market, I would never have guessed that Apple would choose to add a Konfabulator-like feature to Tiger. Konfabulator is so far outside the mainstream of Mac OS X software, so far out in left field, that I'm still surprised Dashboard even exists. I feel for Arlo and Perry since they seemed just as surprised as I was. Does this give them the right to gripe? I think it does; everyone has the right to be upset in response to unexpected bad news.

I'm not sure what Arlo and Perry should have done differently. They created a quirky, fun, user-friendly shareware product that made the Mac platform look good. They were poster children for Mac OS X development. They embraced the unique features of Apple's new OS and used them to do interesting things (e.g., using Quartz to create oddly-shaped translucent windows). While I believe Apple was perfectly within their rights (legal, moral, and ethical) to create Dashboard, I can't help but wonder if the next Arlo and Perry will be discouraged after seeing what happened to Konfabulator.

Apple's creation of Dashboard lit a fire under the previously stagnant effort to produce a Windows version of Konfabulator. It was completed a few months ago, and future Konfabulator development seems to be leaning heavily towards the Windows platform. While I hope budding Mac developers will take a lesson from the resilience of the Konfabulator team, I can't help but think about how Apple's actions have caused a previously Mac-centric development team to spread its talents to other platforms.

In the grand scheme of things, a widget engine is small potatoes. I just hope that Apple remains aware of the consequences of its actions. It's not always easy to differentiate between the inconsequential widget application and The Next Big Thing.

The product

Dashboard has visual frills and thrills galore. In addition to the Exposé-style activation, there's an extremely cool water ripple effect when a new widget is activated. Closing a widget causes it to be sucked back into the tiny "x" button like a piece of transparent taffy. The "widget drawer" on the bottom of the screen, activated by the slightly larger "x" button in the lower-left corner of the screen, pushes the entire screen background upwards as it appears. The widgets it contains slide in from the left. Dragging a widget icon out of the drawer causes it to do a cross-fade "morph" into its full-sized form.

If the Dock was the "ooh, ahh" demo in Mac OS X 10.0, Dashboard is it in Tiger. Here's a screenshot.

Dashboard widgets in action

The ripple effect looks nice in screenshots, but even better in motion.

The resource (memory/CPU) usage of Dashboard widgets is (unsurprisingly) similar to that of Konfabulator widgets that do the same things. Both products will eat CPU time if one or more widgets choose to poll frequently. Thankfully, none of Apple's bundled widgets suffer from this.

Dashboard benefits from using the Web Kit library, which will already be in memory if Safari is running. Unfortunately, it still takes several seconds for widgets to "become live" the first time Dashboard is activated. On the bright side, this means that Dashboard widgets are not running at all (and therefore not using any memory or CPU time) until the first time Dashboard is activated.

The bundled widgets are useful (you can see the full list of widgets at Apple's web site), but many suffer from "widget stiffness"—they are not resizable. This is fine for something like the calculator widget, but the Stickies widget really needs to be resizable if it ever wants to replace the Stickies application.

I have one big complaint. Apple has unnecessarily linked two good ideas in Dashboard: the widgets themselves, and the Exposé-style layer where they live. While I really love the idea of a separate window layer for infrequently used items, I'm extremely disappointed that only Dashboard widgets can live there. To a lesser extent, I'm also disappointed that Dashboard widgets can't be interleaved with regular windows, float on top of all windows, or be embedded in the desktop like Konfabulator widgets can.

I'd like to be able to put any application in the Dashboard layer, and I'd like to be able to pull widgets out of it and place them anywhere I want. This would require rethinking the Dashboard UI a bit (e.g., providing an active menu bar in the Dashboard layer) but I think the benefits would be well worth it.

The Dashboard interface itself is pretty wacky: the widget drawer that seems hidden "under the screen," the weird, twisty "x" button, the strange animations...it's all very odd. Cool, but odd. Even the Dashboard Dock icon is strange. It's the only application that can actually be dragged out of the Dock while it's "running" (i.e., while the little black triangle is under it). Try that with any other application and it'll get sucked back into the Dock. Try it with the Dashboard activation application and it goes "poof" and disappears. Like I said, odd.

The summary

Dashboard is a surprising and interesting addition to Mac OS X Tiger. What happened to Konfabulator is a shame, but I don't condemn Apple for it. I only hope Apple made the choice to develop Dashboard with eyes wide open. When I read about Apple executives trying to claim primal ownership of the concept behind Dashboard, I wonder. Then again, it could just be PR fluff and damage control.

The classy thing to do would have been for Apple to at least acknowledge Konfabulator in some non-legally-binding way. (Please sign these papers indicating that you did not save Itchy & Scratchy, yada yada.) But there was nary a mention of Konfabulator from Steve Jobs when he introduced Dashboard in front of an audience full of developers, many of whom must have immediately made the connection. It was the elephant in the room. To me, Jobs's silence speaks of a guilty conscience.

I expect to see a huge number of useful, fun, powerful third-party widgets produced in the months immediately following Tiger's release. I'm not so convinced about the long-term prospects for the technology. Konfabulator widget development tapered off very quickly after a big initial burst (yes, this was before Dashboard was public). We'll see if the same fate awaits Dashboard.

Even if this widget stuff doesn't pan out, Apple has an easy escape plan: make the Dashboard layer open to any application, or perhaps even individual windows from applications. I'm sure the virtual desktop mavens can think of plenty of other interesting ideas.

Even if you never had any interest in Konfabulator, the Exposé twist of Dashboard may be enough to get you to take another look. At the very least, you can impress friends and neighbors with Dashboard's whizzy special effects. Why a water ripple when a widget is activated? Apple answered the same question about the cube rotation effect for fast user switching in Panther: "Because we can."

The Finder (not really) revisited

Earlier, in the metadata section of this article (it seems so long ago, I know), I wrote about leaving the topic of metadata behind as Mac OS X's metadata features stagnated. At a certain point, everything that can be said has been said, and all that's left is hope, however dim.

With the introduction of Tiger, the Mac OS X Finder has entered this stage of its existence. The Tiger Finder does not include any significant new interface features beyond those added in service of Spotlight, and as we've seen, even those suffer from the now-expected Finder malaise.

Over the years, the Mac OS X Finder has gained a well-deserved reputation as the least pleasing bundled Mac OS X application. It has been called the most widely used piece of abandonware on the Mac platform. While some people like it, few love it, and many hate it.

In the Macintoshian Achaia forum here at Ars, the abbreviation "FTFF" is now used without explanation. It stands for "Fix the F***ing Finder," and it shows up often in Mac OS X wish-list threads. (And if you think my long-running criticism of the Finder makes any Ars forum opinions inherently biased, you don't know the Mac Ach.)

To a casual observer, this might appear a bit extreme. The Mac OS X Finder seems, if not glamorous, then at least benign. What's the big deal? The bad feelings about the Finder don't spring from a single source. There are at least three distinct threads of Finder dissatisfaction, usually appearing in combinations of two or more in any given Finder malcontent.

First, there's the whole spatial/browser issue which I covered extensively in my Panther review. I also wrote an entire article on to the topic. I'm not going to revisit the issue here. Suffice it to say that what I wrote about the Panther Finder in 2003 is just as applicable to the Tiger Finder today.

The second issue is performance. The classic Mac OS Finder was no great shakes when it came to remaining responsive under heavy load or during network activity, but standards were different back then. In the Mac OS X age, the expectations are much higher. Sadly, the Mac OS X Finder has seemed stuck in an earlier era since day one.

While applications like Motion can composite multiple video streams in real time, the Mac OS X Finder UI still chokes when dragging a large number of files from one place to another. Network access remains a notorious cause for application-wide "pauses" (the infamous spinning beachball cursor). Mac web forums are haunted by nightmarish tales of total Finder paralysis in response to accidentally clicking on a QuickTime movie on a networked volume in a column-view window, causing the "helpful" movie preview to be displayed. And iDisk, don't even get me started about iDisk, the poster child for inexplicable slowness. While third party WebDAV clients read and write files on .Mac iDisks just fine, the Finder seems to revel in long pauses and glacial data transfer rates.

Finder performance has improved steadily since Mac OS X 10.0, sometimes dramatically so. But going from "unusable" to merely "frustratingly slow" is not a point of pride. It's not so much the actual magnitude of the problems. Okay, sometimes it is. But mostly it's that the Finder suffers so much in comparison to its contemporary applications.

That leads to the final pillar of Mac OS X Finder dissatisfaction. It's the little things. Little annoyances are easy to forgive in a version 1.0 product, and this is a valid label for original Mac OS X Finder. But when these little annoyances persist over four years of development despite the relative trivialness of their solutions, people start to get pissed off.

There are too many examples to list, although that doesn't stop people from trying in the seemingly eternal forum threads on the topic. I'll just pick three from my personal experience.

The first is icon grid spacing. The Mac OS X Finder has had a very widely spaced icon grid from the start. While this does ensure that very long file and folder names cannot overlap, many users (especially those with small screens) consider it wasteful when dealing with the more common case of reasonable file name lengths.

Some sort of adjustment for icon grid spacing has been requested since Mac OS X 10.0, and yet it remains unimplemented. This issue is exacerbated by the fact that the classic Mac OS Finder had a simple grid spacing adjustment over a decade ago. (We have the technology! We can rebuild it!)

Next there's the "View Options" palette. It contains a pair of radio buttons that control the scope of the changes made in the palette. There's one radio button for "All windows" and one for "This window only." (Let's ignore for now that the concept of actually changing the view options for "all windows" in the Mac OS X Finder is a cruel joke.) When a user brings up this palette, he does so because he wants to change the view options for the window he's looking at. Maybe once in several thousand uses he also wants to try to make his changes apply to "all windows" in the Finder (whatever that means). But 99.9% of the time, he simply wants to change the view options for a single window.

Infuriatingly, the "All windows" radio button is selected by default most of the time. (Good luck divining what set of conditions determine which radio button is selected by default.) What this means in practice is that virtually every single time a user changes view options for a window, he must first (remember to) change the radio button selection. This gets very old, very quickly, let me tell you.

Finally, there's the "size to fit" behavior of Finder windows, triggered by clicking the green "+" widget in the window titlebar. In theory (and in practice in all versions of the classic Mac OS Finder with this feature), this should cause the Finder window to resize itself in order to show as many items in the window as possible, but without any extra space around them. For a Finder window with just a handful of closely-spaced icons in it, this should produce a window with no active scrollbars.

In the Mac OS X Finder, this feature has never worked consistently. When it doesn't work, there's usually one active scrollbar with about 5 pixels of travel in it. The Finder tries to size the window appropriately, but fails to correctly judge the bounds of the icons it contains. This is not rocket science...

In isolation, these all sound like nits being unnecessarily picked. "Boo hoo, your pet bug didn't get fixed." But the cumulative weight of everyone's nits can't be ignored. Complaints tend to cluster around 10-20 problems that plague a large number of people. So it's not a matter of every person having a different complaint, leaving Apple with no good way to prioritize them.

And speaking of prioritizing bugs, it's not as if these little things have been slipping through the cracks because Apple has been busy adding big new features to the Finder in every major release. Instead, the Finder remains essentially unchanged from release to release (excepting perhaps the integration of whatever new Mac OS X feature actually did get attention this time around) and the little things don't get fixed. Remember, this is over four years of development.

Even the little good will that the Mac OS X Finder did initially garner, thanks to its incorporation of the NeXT-style column view and assorted other browser-style features, has been evaporating in recent years. The honeymoon is ending for browser fans. Increasingly, they're coming to the realization that their browser-related features requests are being ignored too.

I also see a lot of complaints about the how the non-browser features of the Finder interfere with pure file browsing. ("Why can't I force all my browser windows to use list view by default?") Suddenly, all the past hubbub over the destructive browser/spatial mix in the Finder starts to make a lot more sense.

Even more painful for browser fans is the existence of an application like Path Finder. Path Finder is a browser-style file manager developed by a single person that absolutely embarrasses the Mac OS X Finder with its huge feature set. Version 1.0 of Path Finder shipped in 2001 after only 6 months of development. Today's version 3 is miles ahead of that, and the upcoming version 4 is like the Final Cut Pro of file management. It's a file browsing tour de force.

The Mac OS X Finder, Tiger's included, is none of these things, and it's a shame.

The Spotlight factor

Spotlight has been the great white hope for Mac OS X file management. At Apple's 2004 Worldwide Developers Conference, Steve Jobs himself opined that Spotlight has the potential to actually replace the Finder. "A lot of us are never going to use the Finder again," he said. "We're just going to go right here [to Spotlight] to find anything."

I don't want to minimize the value of a fast, powerful, accurate system-wide search facility. It really is a wonderful thing to have, and it handles tasks that the Finder is totally unsuited for. But it is what it is: search.

Replacing all file management tasks with search is simply self-flagellation. Search is great when you want to find a needle in a haystack, but it's unnecessarily inefficient when dealing with a small number of frequently used files. This is why folders exist: to collect sets of files related in ways that have nothing to do with their contents.

Spotlight's smart folders could help here if Apple better leveraged Tiger's new extended attribute features to allow the addition (and indexing) of arbitrary metadata. But as things stand, the task falls to plain old folders.

Given a "working set" of frequently used files, it's extremely efficient to put them all in an open folder window (maybe in list view with certain subfolders disclosed) and double-click or drag and drop the files in order to open or edit them. The search-based alternative to this straight-forward Finder interaction requires activating the Spotlight search menu, typing the first few characters of a file name, selecting the file from the search results menu (or activating the full search results window if it's not in the menu), then selecting, double-clicking, or dragging the file in order to open or edit it.

This all assumes that the file's metadata is unique enough to make it easily distinguishable in the results list, of course. Pity the poor web developer trying to use Spotlight to edit a series of identically-named and similarly structured "index.html" files, for example.

Remember that this whole process has to be run through from start to finish each time a file is opened. And unlike special-purpose "launcher" utilities like Quicksilver, Spotlight will not "learn" based on your actions. In fact, it won't retain any state at all beyond, perhaps, the last search string entered. Each search essentially starts from zero.

If this still sounds like a good way to work with your personal "working set" of files, I challenge you to try it for a month and see how it feels. Don't use the Finder at all; try to do everything through Spotlight. Who knows? Maybe your "working set" really is a random distribution of all the files on your hard drive. If that's the case, then Spotlight will be like manna from heaven.

But if you're like most users, repeatedly "searching" for files you have opened five times already today—files that are likely collected in a handful of folders—will slowly drive you mad. This is doubly true if you are a developer working in an environment where file names are frequently identical or where the hierarchy is part of the product (e.g., a web site or a Perl module distribution).

Again, I don't want to marginalize the power of search. Search is great; search is important. When you're looking for that one obscure file you just know is on your hard disk somewhere, or even just looking for a word or phrase you think you saw once, Spotlight kicks ass. But a replacement for the Finder it is not.

In Mac OS X, the increased use of search is kind of a self-fulfilling prophecy. The more the Finder sucks, the better Spotlight looks in comparison. But no matter how bad the Finder has gotten, there's one feature that has persevered: the desktop. As the most spatially consistent part of the Finder, it's the one "location" in the computer that virtually every user feels comfortable with—which is why so many people use it to store their most important or frequently used files (much to the chagrin of computer neatniks everywhere).

Applications like iTunes and Mail have already demonstrated the power of a proper mix of search and traditional, structured information management. The search field in iTunes is great, but iTunes would be a lot more frustrating to use without the hierarchy provided by the the genre, artist, and album browser panes. The same goes for Mail with its fast search field in addition to traditional mail folders.

In both cases, the hierarchy is really just a set of canned searches. That's also (effectively) the case on volume formats like HFS+ where the file path is just one possible unique identifier for a file. Plain old folders are really just "canned searches" based on file path metadata. The resulting interface is the important part, not the mechanism.

I hope that the combination of the Finder and Spotlight can some day live up to its potential. In the meantime, it's a mistake to overemphasize the utility of Spotlight as a means to cover up for the Finder's shortcomings. Spotlight is great, but it doesn't entirely eliminate the need for traditional file management. The Finder's not off the hook.

Internal changes

There are some internal changes to the Tiger Finder that are worth mentioning. The first is the modularization of the Finder's "copy engine." Applications other than the Finder often want to move or copy files. This is a surprisingly complex task, especially in Mac OS X with its Mac-specific metadata, resource forks, symbolic links, hard links, and complex rules for the emulation of all of these features on volume formats that don't support them.

The classic Mac OS solution was for applications to simply ask the Finder to move or copy files on their behalf. This worked, but it made all of these applications dependent on the Finder. During the introduction of Mac OS X, a lot was made of the fact that the Finder was now "just another application." This was partly a reference to classic Mac OS applications' historic reliance on the Finder for file management tasks. Mac OS X was to be different. Applications could no longer simply assume that the Finder was always running and available. If a Mac OS X application wanted move or copy files, it'd have to do it on its own.

In Tiger, Apple has put the Finder's file copy engine into a public library that any application can use. The Tiger Finder itself actually uses this library. This unification of "correct" file operations into a single library is an important step, especially with the potential proliferation of file metadata in the years ahead.

The other notable change to the Finder comes thanks to a combination of file system notification APIs introduced in Panther (but not leveraged by the Finder at that time) and the features added on as part of Spotlight. The Tiger Finder now correctly shows files when they are created and makes them disappear when they are deleted.

If this doesn't sound like a big deal to you, then you probably haven't used the Mac OS X Finder very much. Or maybe you simply have low expectations for your file manager. (A Windows user perhaps? F5!) But you know how crazy we "graybeards" are, always doggedly holding onto the quaint notion that the file manager should actually reflect the state of files on disk in something approaching a timely manner.

The Mac OS X Finder has historically failed miserably in this area, usually refusing to show files created outside the Finder until the user explicitly clicks in the window. In theory, applications are supposed to notify the Finder when they create files. In practice, many don't—and many can't, like Unix command-line tools that have absolutely no knowledge of the higher-level APIs in Mac OS X, let alone the Finder.

Well, those days are over. Thanks to the kernel hooks that make Spotlight so magically responsive, the Tiger Finder can no longer be surprised. It reflects file system changes instantly, regardless of their source. Behold!

I firmly believe that this change alone will save me more aggravation than all of Tiger's other improvements combined. If a Mac OS X kernel developer walked by me right now, I'd give him a big hug.

Finder summary

I know I said earlier that I was leaving the Mac OS X Finder quagmire behind due to the application's stagnation and my past exhaustion of the topic. Rest assured, what you just read was the short version. But I did want to try to explain why the issue refuses to die completely. The dissatisfaction with the Mac OS X Finder is real, and it doesn't show signs of going away any time soon.

I covered the slightly disappointing implementation of smart folders and the wacky Spotlight integration earlier, but it bears repeating: seemingly anything that comes in contact with the long-neglected Finder ends up, at best, wallowing in mediocrity.

I haven't given up on the Finder entirely. If the Mac OS X metadata saga has taught me anything, it's that vigilance at least has the possibility of paying off in some small way. I just wonder how many more years I'll have to wait before the Mac OS X Finder finally bubbles up the priority list at Apple.

Despite all of this, the Tiger Finder is better than its predecessors. Its newfound awareness of file system changes alone is worth the price of the upgrade. Just try not to think about what might have been.

Performance

In many ways, the performance story in Tiger is much the same as it was in Panther, so much so that I'm going to be lazy and paraphrase what I said then.

For over four years now, Mac OS X has gotten faster with every release—not just "faster in the experience of most end users," but faster on the same hardware. This trend is unheard of among contemporary desktop operating systems. It certainly didn't apply to classic Mac OS, where every significant new OS revision was perceptibly slower than its predecessor on the same hardware. (System 7 and Mac OS 8, I'm looking at you.) The world of Windows has historically followed a similar trend. It is usually taken for granted that a new OS will not really sing until you upgrade your hardware.

This is has not been the case with Mac OS X. Every single major revision of Mac OS X has been faster than its predecessors when running on identical hardware. Jumping several major revisions is practically like getting a new machine. It's impressive, and unprecedented.

Of course, the pessimistic angle is that Mac OS X's performance was so abysmal in version 10.0 that it had nowhere to go but up. Like System 7 and Mac OS 8 before it, Mac OS X is indeed perceptibly slower on the same hardware than its distant predecessor, Mac OS 9.

But that's really reaching. At a certain point, you simply have to accept that Apple is doing a heck of a lot more than patching up grossly inefficient code rushed out to make the 10.0 release. It's time to give credit where credit is due. Apple keeps making Mac OS X faster, and it rules.

Interface responsiveness

At this point in the performance section of my Panther review, I took a depressing turn into the world of interface responsiveness in Mac OS X. Long derided for its comparatively glacial response to scrolling and window resizing, OS X has continued its search for "teh snappy" (as Mac forum minions cheekily misspell it; I will not follow suit here).

Each major version of Mac OS X has been hailed as the future bringer of "the snappy," and each actual release has caused a debate about the reality of its arrival. "Argh, window resizing is still too slow!!!" "Works great here. There must be something wrong with your Mac." "I'M NOT SHOUTING YOU'RE SHOUTING!" "NO YUO!" Etc. etc.

I suppose the fact that the issue continues to come up before every major OS release is strong evidence that a truly responsive interface hasn't really "arrived" to everyone's satisfaction, but things have certainly improved with each release.

Tiger takes the biggest leap yet in this area, and that's saying something. Improved window resizing responsiveness in particular is immediately noticeable in Tiger. It's still not perfect, but it's a big improvement. Compared to Jaguar or earlier it's like night and day. Scrolling is also a bit snappier, although this is not as noticeable, perhaps because scrolling was already pretty good in Panther (in most apps, anyway).

It would be easy to attribute all of these improvements to Quartz 2D Extreme. It'd also be wrong. Disabling Quartz 2D Extreme (using the Quartz Debug application that's part of the free developers tools bundled with Tiger) has little perceptible effect on window resizing, in particular, for most applications. What this means is that not all UI performance is gated by drawing speed.

It appears that window resizing is much more likely to be held back by the CPU overhead of iteratively reflowing/refreshing UI elements—especially nested elements—within the window than by the time required to do the actual drawing. This is part of the reason that applications with complex interfaces like iCal (many panes, buttons, lists, and cells) resize so much more sluggishly than applications with simpler windows like TextEdit.

Some applications mitigate this effect by neglecting to reflow certain UI elements until the resize operation completes. Mail, for example, does not reflow its message list columns until you release the mouse button after resizing.

Scrolling does tend to be more drawing-limited than resizing, but a fast CPU can hide a lot. Remember that software drawing performance has been dramatically improved in Tiger. But far fewer applications are gated by actual drawing performance than you might imagine. The upshot is that it's very difficult to even notice when Quartz 2D Extreme is turned off (on either the dual G5 Power Macs or the G4 PowerBook) during casual use.

Spotlight performance

There's one slightly dim spot in the Tiger performance picture, and unfortunately it's going to be a part of every user's first experience with the new OS. When booting into Tiger for the first time, Spotlight has to index all attached drives. This is done with a low-priority background task, but there's no avoiding the huge amount of disk activity involved. Expect your first few hours in Tiger to be accompanied by the steady tick and pop of all of your hard disks.

The performance impact of indexing is negligible on any operations other than disk access. The CPU is not swamped and graphics draw just as fast as always. But launching applications or doing anything else that requires disk i/o does slow down noticeably. There'll be an increased cacophony of hard disk noises and many more bounces of Dock icons.

This effect was exacerbated by the slow external disk that I used to test many of the pre-release versions of Tiger, so maybe it got stuck in my head more firmly than it deserved to be. It's not as bad with modern desktop hard disks. The PowerBook's lower-RPM internal hard drive didn't fare as well, but then I suspect Mac laptop users are used to slow disk access at this point.

Anyway, it's all over in a matter of minutes or hours, depending on the size and number of your hard drives. Once Spotlight is all caught up, it stays up to date though that kernel magic that we love so much. This on-the-fly indexing in response to individual file operations was imperceptible to me, even when running off a very slow disk. Spotlight appears to index only the first 10MB or so of file contents by default, although metadata importers can choose to ignore this limit.

Performance summary

There's not much more to say. Tiger is faster than Panther, and you'll notice. The GPU-powered graphics technologies play less of a role in day-to-day performance increases than you might expect. Think of them instead as enablers of entirely new things (e.g., Core Video effects) rather the bringers of "the snappy."

Has the interface responsiveness everyone has been clamoring for finally arrived in Tiger? Let me put it this way: I fully expect to see our friend "the snappy" appearing in Mac web forums during the run up to the Mac OS X 10.5 release in late 2006 or so. (Resizing an application like iCal is still way, way too slow.) But Tiger pushes things much closer to the tipping point.

Grab bag

Tiger is a big OS release, and this review could go on for ever. (I know, I know...) Apple touted Panther as having "over 150 new features." Tiger apparently has "200+." You can easily spend a day just reading through the Mac OS X section of Apple's web site. There's even a page that actually lists all 200+ new features. I encourage you to read it because I'm certainly not going to cover them all.

This article is in the home stretch now, but there are a lot of little things that I'd like to talk about briefly. I'm calling this section the grab bag. It'll contain a series of brief (I hope) trips into individual applications and features, one after the other with no real transitions between them. Think of them as the Ars equivalent of Perl's lightning talks.

If your favorite topic ended up in the grab bag, I apologize. But, honestly, it's not possible to cover everything in a huge OS release like Tiger to the depth that I covered, say, metadata or Quartz. I chose to go deeper into the topics that are of the most interest to me, and that I feel are the most significant, technologically. Opinions will obviously vary. I encourage readers to expand on the topics that interest them in the discussion forum.

Enough setup, let's begin...

System Preferences

System Preferences uses the new unified titlebar/toolbar look. It has a new search field in the toolbar area. Typing a query there dims the entire window and then highlights the preferences panes that have something to do with the query using a "spotlight" effect. (Get it? Har.)

The System Preferences search field in action

I'm not sure if System Preferences actually uses Spotlight to do the search. It's conceivable that it could, but that's a bit like using a sledgehammer to kill an ant. SearchKit, a Panther technology, really provides all the required functionality for this feature. The graphical spotlight effect is, of course, entirely custom. Ah, marketing.

The query matching system is helpfully broad, accepting many synonyms for the various bits of functionality controlled by preference panes. For example, the search strings "wifi," "wireless," "airport," and "802.11" will all lead to the Network preference pane. Even Windows-centric terms like "wallpaper" are properly mapped.

Unfortunately, in an apparent effort to highlight the new search features, the customizable toolbar is gone. In its place is back/forward navigation and a "Show All" button. For those of us who know where we want to go, this new UI is frustrating. Switching from one preference pane to another requires clicking the "Show All" button first, then selecting the new preference pane (or using the menu...yuck). Meanwhile, the toolbar area remains almost entirely empty; the search field doesn't even expand to fill all the dead space. Here's hoping the customizable toolbar makes a comeback in 10.4.x.

Sync Services

Sync Services is a new framework for synchronizing data between applications, devices, and entire machines. Think of it as the publication of the functionality behind iSync to all third-party developers. The iSync application is now just an interface to the public Sync Services engine. In fact, it actually passes off all machine-to-machine syncing to a tab in the newly enhanced .Mac preference pane.

The .Mac sync preference pane

I expect a lot of third party developers to add sync support to their applications. Apple has already expanded the reach of Sync Services among its own applications, adding support for syncing Keychains and Mail accounts, rules, signatures, and mailboxes.

The interface for developers is very nice. They don't have to worry about many of the ugly details of data syncing: conflict management, change notifications, duplicate data detection, or .Mac connectivity.

That leads to the one sore point. Applications that use Sync Services automatically support .Mac syncing, but there's apparently no facility for syncing with networked services other than .Mac. It would be nice if the network storage part of Sync Services was also expandable by third parties. But I guess this is yet another way to make the $99-per-year .Mac subscription service more attractive.

Core Data

It seems like a lot of Mac enthusiasts are excited about Core Data. Unfortunately, many of them have it confused with Spotlight (which, in turn, is often confused with arbitrarily extensible file system metadata). Apple's partially to blame here—too much technology branding makes everything start to blend.

Core Data is actually a technology for developers. It's a framework for object management and persistence. It allows a developer to define object attributes and relationships, then build an application based on this data model without having to write any code to manage the object graph or serialize objects. The currently supported serialization formats are XML, binary (essentially compressed XML), and SQLite. Applications that use Core Data also get niceties like undo/redo for free.

If none of that made any sense to you, that's okay. You won't be using Core Data unless you're a developer. Furthermore, users will probably not be able to distinguish an application that uses Core Data from one that does not. If your interest is piqued anyway, you can read more about Core Data on Apple's developer web site.

Core Data should mean better Mac applications in less time. But like all frameworks that try to do a lot for the developer, Core Data also imposes some constraints. While it may be a godsend to developers writing applications that fit well into the Core Data world view, Core Data is not a panacea. And don't expect the big developers (Microsoft, Adobe, Macromedia) or anyone else with a substantial existing code base to pay it much mind—especially if the application is cross-platform.

Scalable User Interface

In Tiger, Apple has laid the groundwork for a user interface that can scale from current ~100dpi screens to future screens with much higher pixel densities. The Quartz Debug application bundled with Tiger's free developer tools can be used to adjust the scale of the user interface in Tiger.

Quartz Debug's experimental user interface resolution palette

Some applications will look strange (or downright broken) when scaled in Tiger. Most remain usable, however. Text and other vector-based graphics look good when scaled up, but low-resolution bitmaps do not. The scaling is per-application, so it's possible to mix and match windows with different scale factors.

Safari at 2x next to a normal window. Click for a full-sized view.

It's kind of hard to tell what's going on in the reduced screenshot above. Check out the full-sized screenshot to get a better idea of how it looks. The Safari window is scaled by a factor of two and the user interface resolution palette is at its normal size.

Needless to say, regular users are not expected to use this feature. The user interface resolution adjustment exists in Tiger so that developers can prepare their applications for a future version of Mac OS X where this feature will be an integral part of the OS.

When that time comes, Apple will have to update all of its applications as well. The resolution of all the user interface widgets and textures in Mac OS X will also have to be increased substantially. Like all bitmaps, the current artwork looks ugly when scaled up. In a future resolution-independent version of Mac OS X, Apple will have to use much larger bitmaps, ensuring that they will only be scaled down, never up.

Mac OS X's use of 128x128-pixel icons seemed excessive back in 2000, but even that size has proven to be a bit small. Apple has quietly expanded the maximum supported dimensions of Mac OS X's "icns" icon format to include 256x256-pixel icons, and 512x512-pixel icons are not out of the question for the future.

Which will come first, the affordable 300dpi display or the resolution-independent version of Mac OS X? The race is on.

Safari

My Panther review contained a paean to the glory of Safari. The new version (2.0) continues those winning ways. Tiger's Safari comes with the expected (but still appreciated) collection of page rendering performance improvements and better conformance with web standards. The performance of the JavaScript in particular has improved. This makes script-heavy web application like GMail feel a lot snappier. (As a bonus, Panther users get most of these improvements in Safari 1.3 which is part of Mac OS X 10.3.9.)

Safari 2.0 supports RSS in an interesting way. Articles from one or more news feeds are displayed in a single web page. A slider control adjusts the length of the article summaries. An example is shown below. You can see it in motion at Apple's web site.

Safari 2.0 displaying an RSS news feed

Apple considers this feature so significant that the new version of Safari is branded as "Safari RSS" in Apple's marketing literature. I think that's pretty silly. First, because Safari supports Atom in addition to RSS, and second, because of the proliferation of quality news readers on the Mac platform these days. I'll give Safari's news feed support a try, but I have a hard time imagining it pulling many people away from NetNewsWire and friends. Perhaps it will raise the awareness of news feeds in general among Mac users.

The other important new addition to Safari is a comprehensive search feature for bookmarks, news feeds, and browser history. Safari doesn't index the contents of web pages like OmniWeb 5 does, but at least searching my browser history no longer involves manually scanning a bunch of giant submenus.

Speaking of browser history, there's a convenient "private browsing" feature that prevents browser history, downloads, and form entries (e.g. usernames and passwords) from being recorded by Safari. This is helpful when using another person's Mac, although other uses have been suggested...

Finally, Safari has converted some kinds of error messages from slide-down "sheets" to pseudo-web-pages.

Safari's new error message display

I find the new errors a lot more pleasant than the old sheets, if only because they display faster and do not demand to be explicitly dismissed before I can do something else in the browser window. I do worry that this path might lead to the "friendly" (but totally uninformative) error messages used by Internet Explorer for Windows. I'm pretty confident that Safari won't make that mistake, however.

Mail

Like Safari, Mail has graduated to the big two-point-oh in Tiger. I already covered Mail's strange new look. There are actually a few new features as well. The most significant is support for "smart mailboxes" which behave like smart playlists in iTunes or smart folders in the Finder. They're basically mail folders that contain the constantly updated results of a search query. Entourage has had this feature for years. I'm glad to see Apple folding it into their free e-mail client.

Mail now saves all of its messages in individual files in order to allow Spotlight to index them more easily. Earlier versions of Mail sometimes (depending on the account time) stored multiple e-mail messages in a single file. I have to wonder if the Mail data folder will become a huge sea of tiny files after a few years of use...and if it does, is there any problem with that? Most file systems tend to get angry when a single directory has more than, say, 100,000 files in it. I hope Mail 2.0 will subdivide the data directory as necessary to stay out of any unpleasant performance tar pits.

Automator

Tiger includes Automator, an application that lets regular users create automated tasks without any programming knowledge. It works by stringing together "actions" which are programmer-created interfaces to the features of other applications. Tiger ships with a large set of Automator actions for many of Apple's bundled applications.

Here's an automation that I created in a few minutes. It adds an audio reading of all my e-mail messages that contain the word "apple" to my iTunes library.

My first automation

To build an automation, simply drag actions from the left pane to the right pane, then configure the adjustable bits. The output of each action is the input of the next. It's really easy, and can be very powerful if what you want to do is covered by the available actions.

Automator is a clever application. I'm not sure how many home users will find their way to it, however. Maybe the cute robot icon will help.

iChat

iChat learns some new tricks in Tiger. Up to four-way video conferencing is now possible. I tried it in an earlier build of Tiger. (I haven't had a chance to try it with the final version, unfortunately.) The video looked increasingly fuzzy as people were added to the conference. It seemed like it wasn't using the full upload bandwidth of my cable modem, but maybe the bottleneck was elsewhere. Anyway, despite the fuzziness, the latency was low. The video was admirably responsive to sudden movements, and the audio delay was minimal.

iChat now supports the Jabber instant messaging protocol, mostly because Apple decided to add an IM server to Mac OS X Server 10.4. Jabber is an open protocol, so there was no AOL or Microsoft to pay for the rights to use it. iChat's new icon reflects its new multi-protocol abilities; the AIM "running man" badge is gone, leaving a slightly bare-looking blue bubble.

The iChat buddy list now supports inline buddy groups (finally, jeez). Multiple account support is improved—unsurprising, given the Jabber support. iChat may also be the first place you'll notice "Bonjour," the technology formerly known as Rendezvous. The name change is the result of a trademark dispute settlement. I liked the old name much better. Sigh.

Grapher

The Graphing Calculator returns to the Mac in Tiger...sort of. The Grapher application (as it is called in Tiger) is not descended from the original Graphing Calculator from 1994. Instead, it appears to be an evolution of Curvus.

Grapher does 2D and 3D graphs, but does not seem to include my favorite feature from Graphing Calculator: draggable algebra. Dragging variables from one side of an equation to the other would automatically do the right thing in real-time as you dragged. It was neat.

And now, a gratuitous graph of an equation that I don't understand.

Some sort of graph...

Dictionary

Tiger includes an electronic version of the Oxford American Dictionary and Thesaurus. There's also a Dashboard widget that (presumably) reads from the same data. The dictionary application is simple, with a search field, back/forward navigation, and buttons to adjust the size of the text. A system-wide keyboard shortcut (command-control-d by default) will look up any selected word in the dictionary.

I'm sure I'll use this application a lot. I used to use OmniDictionary, but it pulled its definitions from a server over a non-standard port, running afoul of most firewalls. Its definitions also never looked nearly this good.

The Oxford American Dictionary

I like the Dictionary application better than the Dashboard widget. The search field is bigger, the navigation controls are easier to hit, and the typeface and colors are more pleasing to my eye. Again I find myself wishing that regular applications could be placed in the Dashboard layer.

The strange case of the "button-menu"

The second biennial "WTF were they thinking?" Mac OS X user interface award goes to the the PDF button in the Tiger print dialog box that also functions as a pull-down menu if you click and hold down. It's a button-menu!

WTF?

The new options for PDF-related printing activities are nice, but why in the world are they sprouting from a button? Why not use, oh, I don't know, a pop-up menu widget instead? Yes, then the dialog would need a pop-up menu and a button, but somehow we'd all survive.

The inaugural winner of this award, the Panther Finder's crazy context menu for labels, is still the overall champion, I think. But Tiger's PDF button-menu gives it a run for its money. (The Mail toolbar gets an honorable mention this year as well.)

Default desktop background image

Tiger includes a new, larger default desktop background image (2560x1600 vs. 1600x1024 in Panther). The increased resolution is no surprise, given Apple's recently introduced 30-inch display. The image itself is subtly different from the previous version. Compare:

Panther

Tiger

To quote a forum post by Thaen, "Someone got paid to change that. Think about it." Actually, it doesn't surprise me. Dissatisfaction with the status quo is what drives all progress. If any technology company is going to indulge its employees' pursuit of perfection, it's Apple.

Yes, the spacing still looks like ass

I am once again sad to report that my pet Apple Type Services bug remains unfixed in Tiger. For all you folks out there who, like me, want to use 9-point Monaco as your Terminal font, the work-around is still the same: manually edit your ~/Library/Preferences/com.apple.Terminal.plist file and set the value of the FontWidthSpacing set to 1.003, a value that is impossible to get via the GUI.

<key>FontWidthSpacing</key>
<string>1.003</string>

Then never, ever touch the font panel in the Terminal application, and patiently wait another year. (Well, probably 18 months this time.)

Conclusion

Tiger is the best version of Mac OS X yet. It offers substantial improvements over Panther in all important areas. The performance improvements are immediately noticeable. Every major bundled application has been improved. There's an unprecedented number of substantial, totally new features and technologies: Spotlight, Core Image and Video, Quartz 2D Extreme, Dashboard, and Automator, just to name a few.

This is all on top of the most significant revision to the core operating system in the history of Mac OS X. Fine-grained locking in the kernel prepares Tiger for future Macs with multi-core CPUs. A stable, abstracted, forward-compatible system for kernel extensions frees Apple to make more kernel changes in the future without worrying about breaking existing kernel extensions. And as usual, nearly every bundled Unix program has been updated.

For the first time in the history of Mac OS X, Apple has made progress in the realm of file metadata. While Spotlight plays it safe with external index files tracking information gleaned from existing file system metadata and file contents, Tiger's extended attribute APIs lay the groundwork for a much more interesting future. Uniform Type Identifiers add another missing piece to the metadata puzzle. If Apple can fully realize the potential of these budding, and still disparate systems, Mac OS X may finally catch and surpass the previous file metadata technology leader: the venerable BeOS. My fingers are crossed.

If you're still running Jaguar or earlier, you really owe it to yourself to upgrade to Tiger. It'll be the best $129 you've ever spent on an operating system. If you're happy with Panther, I strongly recommend going to an Apple store and checking out Tiger in person. Chances are good that there'll be at least one or two features that you'll decide you need, if not right way, then soon. As with any new release, it won't hurt to wait for version 10.4.1 or later.