Omar Roth

The value of "AI" is mostly finite

2026-05-10 11:57:01 -04:00

The amount of hype around "AI" and software tooling is pretty exhausting at this point. For each new model, there is a wave of excitement announcing that it's either going to hack the planet, make human labor obsolete, or kill everyone.

However there's a problem, and it's three-fold: most of the output of these models (specifically for software) is low-value, there is a finite amount of this low-value software that can be created using these tools, and usage of these tools will slow once that software has been produced.

To be clear, I am not claiming this stuff is worthless. However, as more and more of the actual cost of using these tools is passed onto the consumer, the value produced will likely not be enough to recoup the amount of investment into these new tools.

# The Slop Trough

Since software development is currently the most trendy area of investment for new models (besides cybersecurity), I'm going to focus specifically on that industry, but note that this argument applies to almost every other area currently being disrupted by this technology.

Every software product has a cost to plan, build, and maintain. You can imagine it took some amount of effort to produce each line of software, and for some amount of effort expended, there is some amount of value produced. It is obviously possible to produce software "at cost", where the amount of expended effort outweighs the produced value. This can sometimes happen in the hobby sphere for "personal itches" or for subsidized growth to gain market share (hint hint), but generally this does not result in a profitable business. An easy way to visualize it is like this:

Note that the X and Y axes are labelled in more abstract terms, since "value" can be measured as "does this scratch a personal itch" instead of as "how much would people buy this for if I were to sell it".

The rise of LLMs has added a new line where there now exists something non-human that can still produce value. The cost of running a model is non-zero, but (in theory) it can produce equivalent value as a human with less effort:

One thing you'll notice is that the line for "LLMs" tapers off. What I've seen both from using these tools myself and from watching others, is that there's a point where the amount of value produced with these models levels off. In other words, it takes so much work to prompt a model to do the right thing that it would be better to have a human do it. This effectively means there is now a finite region where work done by LLMs is actually "profitable". As in, a model produces something with the same value as something done by a human but for less cost. It's disputable where the "effort" line is actually drawn (especially when usage is heavily subsidized) and where exactly it levels off (since better models will likely improve in quality over time), but you can make the same argument based on the following assumptions:

Current frontier models are not as capable as humans
There is a cost to running these models (I'm only considering inference, but training will likely also be an ongoing cost)
There is a cross-over point where a model stops producing meaningful value even when expending additional effort (the marginal value decreases)

What this means is there is a "slop trough", where valuable software that previously would not be cost-effective to build, is now being rapidly produced using these tools:

My argument is that the largest amount of tokens will be consumed when building that newly-profitable software, not in maintenance. Once all that low-hanging fruit has been picked, unless there's substantial maintenance required for this software, or other markets can be captured, it's unlikely to lead to growth in the usage of LLMs.

Additionally, all the software that is being produced will be inherently lower value, since otherwise it's much more likely that it would already have been built.

# So what are people actually building with this stuff?

The easiest way to see the leveling off of value is simply to look at what some of the biggest proponents of these tools are actually building.

Probably the most well-known are from internet lolcows like Garry Tan who, in the last couple months, has shipped about 37,000 lines of code per day. This has resulted in some incredible tools like gstack, a suite of employee-like AI-agents for producing more "AI" shit. Or gbrain, a set of text files that can teach LLMs to do things like: search the web, wire into other vibe-coded projects, or schedule meetings (similar to Google Meets). The problem is that, unless there's some other knowledge moat or IP, the major differentiator is simply scale, or the amount of prompting someone has done to produce the desired output.

My CTO friend texted me: "Your gstack is crazy. This is like god mode. Your eng review discovered a subtle cross site scripting attack that I don't even think my team is aware of. I will make a bet that over 90% of new repos from today forward will use gstack." https://t.co/P7aOFu5wFM
— Garry Tan (@garrytan) March 12, 2026

Since usage of frontier models is currently heavily subsidized (some numbers suggest by up to 8x the cost of a subscription), there is a heavy incentive to burn tokens for producing software that is near worthless (see above). Once the actual cost is passed on to consumers and businesses, the size of the trough has to shrink. The problem doesn't change when using self-hosted models either: making the cost lower (the trough "fatter") pushes down heavily on the quality of output produced.

Note: There are plots of public models tracked by price versus performance. What's interesting is all the most "efficient" models are from VC-backed labs (and the price per million tokens is a logarithmic scale but the ELO score is linear).

From my own experience, the best usage of these tools is for "care and feeding" tasks: things like CI, packaging, some types of documentation, and small "throwaway" pieces of software for filling specific needs. The cost of $40/month is maybe valuable enough to cover building an initial prototype, but after that point (and especially if the subscription cost goes up) I will not pay for additional usage.

The biggest wins I see are from non-programmers able to quickly develop tools for personal usage (the rise of single-person software), for making senior developers more productive, and for the above mentioned "maintenance" types of work. For those types of tasks, there will be a large upfront cost of developing applications that I expect to dry up. I've written previously that it's now easy enough to bother giving my website a facelift, but now that's done and I have not used an LLM to that extent since that original rewrite.

# Given enough tokens, some bugs are shallow

With the recent claims from Anthropic finding multiple bugs using Mythos and putting vulnerability researchers out of work, we can look at cybersecurity for another example of how new tooling might eliminate lower margin work. I think it is a good example especially because this has happened before with the rise and widespread use of fuzzing.

Put simply, fuzzing is the process of feeding random inputs into a program to see if it causes unexpected behavior. There are many, many variants of this idea to find different bugs, but even simple fuzzers have proven to be unreasonably effective. To improve the security of commonly used software, large amounts of compute have already been donated to run these tools against commonly used open-source software to find bugs.

The problem is that there are certain types of bugs a fuzzer cannot find, and once it's been run against a piece of software, the "fuzzable" bugs are found and fixed. This has lead people to develop better, more efficient strategies to find more and more obscure bugs for smaller and smaller wins. It is therefore a major breakthrough when a better technique is discovered. I expect there will be a similar sort of "divination" where people try to make their "AI-researcher" more effective through "prompt engineering" or better guidance from skilled researchers.

Combining the results from models with human insight is one of the most profitable approaches (see: two of the recent high-severity vulnerabilities in Linux) but it also isn't new: this is how people currently use fuzzers to triage bugs in popular software!

The funny part is that even though fuzzing is effective and (relatively) cheap to setup, a huge amount of code in the world still isn't being fuzzed! This indicates that even though the cost is low, it isn't low enough for all software. The case is similar with many other types of sanitizers and static analysis tools that have a much lower cost than using models to find novel bugs.

Use of these new tools makes it possible to scale analysis to more and more code, but there will still be a "long-tail" of code that is not important or "profitable" enough to run LLMs against. As all the easy wins from running against critical software are patched, there will still a be need for human insight to identify areas that are potentially vulnerable and develop new approaches (this was already the case for CopyFail and DirtyFrag).

As a side-note, I expect one of the best uses of LLMs will actually be from simply identifying cases where existing fuzzing infrastructure is either not used or is insufficient in finding vulnerabilities. Also, unless it's cheap enough to re-run an "AI-audit" against an entire codebase, once a project has been checked, it's likely that only new code will be checked to see if it contains any potentially exploitable vulnerabilities. That does not indicate growth.

# The consumer market will not backfill demand

The reason for focusing on software is that it appears to be the highest possible margin usage of these tools. As that market fills out, it seems obvious that the consumer market would be the next, much larger target for growing demand in these tools. For things like design and creative writing (among others), that is likely not the case.

The average person uses these tools as summarized, ad-free search. If a voice-assistant like Siri or Alexa were to cost $200/month to use, the average person isn't going to use it. Consumers are extremely price-sensitive. OpenAI is likely able to claim 900 million weekly active users because it's free and provides better results than Google. If they were charged the actual cost, people wouldn't use it!

The reason why software developers are able to afford paying $2,000/month on tokens is because they're extremely high-compensated workers producing roughly comparable output to one another. This means anything produced that is at all comparable to something produced by an engineer will have inflated returns compared to any other industry. Especially markets where the profits are heavily lopsided in favor of the top 10%, it's unlikely that LLM-produced output near the bottom has any actual value.

# There won't be a one-person unicorn

A simple question for anyone who says that models can produce high-value software that wouldn't exist otherwise: if AI were able to produce software that was both highly valuable, and possible to make at a low cost, why wouldn't someone have done it already?

Where is the one billion dollar company with a single founder? It paradoxically requires two things: someone has to be able to produce a high-amount of value with a model, but for some reason not valuable enough that it would've already been done. The types of businesses AI enables are inherently low-margin. These are businesses like call centers, not the next Amazon.

Note: MEDVi has recently made news cycles with claims of being the first company to reach a $1 billion valuation primarily by selling alternative GLP-1 medication. As the regulatory environment tightens, the opportunity exploited by MEDVi and similar companies will likely disappear. MEDVi has already received an FDA warning for misleading advertising of compounded drugs, and there have already been patent infringement suits against prescribing physicians. This appears to be a company exploiting a temporary opportunity created by a loose regulatory environment rather than a new business enabled by AI.

# Looking forward

Most of these observations aren't particularly clever, so I expect OpenAI and Anthropic are keenly aware of these problems internally and how to mitigate them. Especially as the total capital expenditure for AI build-out is approaching (or has already passed) $1 trillion, finding new sources of revenue or figuring out how to capitalize as much as possible on current usage is top of mind.

Since billing based on actual usage will receive more and more pushback, I expect the cost of running these models will be paid for by tapping other sources of revenue (instead of VC-funding). The most obvious for the consumer segment is through advertising, although I don't know if that's enough to support a free tier. More likely it will be a Hulu-like model where the base subscription is still partially ad-driven, with higher tiers reducing the amount of targeted suggestions. As mentioned above, there is a sharp limit to how much consumers will likely pay for this type of product, probably close to $5/month unless they are using it in a professional context, which then gets into the problems with enterprise.

In the case of enterprise, models will likely be tuned to increase lock-in as much as possible (potentially by attempting to make the resulting code uninterpretable to both humans and other models) and to make software "maintenance-heavy". Tokens will likely be separated into separate "tiers" to fill the gap in reduced usage: one tier for building software, and a more expensive tier for repairing it.

For the current players, there will be an incredible incentive to make sure that any software produced by a model will have to be continuously maintained as much as possible, preferably by the same model. The big push for AI models as superhuman bughunters might play into that strategy: insert bugs into the output to produce buggier or harder to understand software, then pay a "code review"-bot to identify and fix those issues.

"Token inflation" has already been suggested as a way to increase usage as much as possible by making bots "chattier" to burn through tokens faster. If it's possible to simply produce more output from the same model for the same cost, that's another easy way to increase revenue!

The final option is simply to hook people on these tools to capture some portion of the revenue that would normally go to people producing value. The most concerning demographic for this effect is junior developers. I would expect to see chatbots discounted very heavily to help capture this segment and atrophy the skills of an incoming talent pool. Making the next generation of software developers completely dependent on these tools will keep usage high.

# Final thoughts

As for the solo developer, now probably wouldn't be a bad time to become a freelancer for people or businesses that normally don't need much software. Even if it's still low-value, there's absolutely money to be made through filling that trough.

Hopefully I've made the case that although there absolutely is still value in this technology, it is not the job-killer it has been made it out to be. The type of work that can be produced (almost by definition) will still firmly remain in the "low-value" category, although things have certainly gotten much better than the pure slop from a year or two ago. As models improve, it remains to be seen if they can outgrow the current trough fast enough to recoup the growing investment.

Wayland set the Linux Desktop back by 10 years

2026-03-18 07:40:32 -04:00

Wayland has been a broad misdirection and misallocation of time and developer resources at the expense of users. With more migration from other operating systems, the pressure to fix fundamental problems has become more prominent. After 17 years of development, now is a good time to reflect on some of the larger promises that have been made around the development of Wayland as a replacement for the X11 display protocol.

If you're not in this space, hopefully it will still be interesting as an engineering post-mortem on taking on new greenfield projects. Namely: What are the issues with what exists, why can they not be fixed, what do we hope to achieve with a new project, and how long do we expect it to take?

# Background and the problems with X11

If you're already familiar with X11 and Wayland feel free to skip to the next part.

For people not familiar with Linux, here's a quick rundown of the terms in this space, roughly in the order of highest-level to lowest-level:

Applications
- These are the things you want to run
- Examples: Chrome, Steam, OBS
Desktop Environment (DE)
- This is what manages things like window theming, notifications, task bars, etc.
- Examples: KDE, GNOME
Compositor
- This is what layers windows on top of each other and does animations, graphical effects, etc.
- Examples: Compiz, KWin (for KDE), Sway, i3
Display Server
- This is the thing that manages the display, it also abstracts away some of the hardware details so all the above work on NVidia, AMD, Intel, etc.
- Examples: X11, Wayland
Kernel / Operating System
- The lower-layer thing that manages hardware resources, for us this is Linux
- Examples: Linux, FreeBSD

The above is not a complete list, but it's enough to give some framing for understanding that X11 is a fundamental piece of most Linux environments.

X11 is currently still the most common popular display server in the Linux ecosystem. It was developed in the mid-1980s and, as legacy projects tend to do, has accumulated functionality that makes it difficult to maintain, according to the developers.

So, in 2008, Kristian Høgsberg started the project that became known as Wayland. Wayland (in theory) replaces the display server, as well as some parts of the compositor and desktop environment with a simpler display protocol and reference implementation. The original conceit behind Wayland is to only implement what is needed for a simple Linux desktop. The original implementation was a little over 3,000 lines of code.

Sounds great!

# So what happened?

It's 2026, and Wayland has reached a market share of around 40-50%, or closer to 50-60% depending on your source. I would argue a product that has taken 17 years to gain substantial marketshare has issues hindering adoption. Compare the development of Wayland to a similar project for managing audio: PipeWire. Within ~8 years, every alternative has been mostly replaced. It's been adopted as the default in Ubuntu since 22.04, roughly 4 years after it first launched!

These are the most common issues I have seen from the perspective of a user, so I will try to stay light on what I consider to be mostly irrelevant technical details and instead focus on larger issues around the rollout and design of Wayland.

# Wayland is more secure (which means I can't do anything)

The reason I use Linux and other Unix-likes is that they give me the ability to do whatever I want on my system, including making mistakes! So why is my display server telling me that certain applications that I installed and chose to run aren't allowed to talk to each other in the name of security?

There are multiple cases of this: OBS can't screen record (it segfaults instead), I can't copy-paste, and I can't see window previews unless everything implements a specific extension to the core protocol.

The actual "threat model" here is baffling and doesn't seem to reflect a need for users. Applications are blocked from seeing each other's windows, but they're not able to interact in any other way that could potentially cause problems?

I also don't care for the "security" argument when parts of the core reference implementation are written in a memory-unsafe language. To be clear, I am not saying that software written in C is bad, I'm specifically calling out that making a security arugment about software then repeating decisions of the previous (40-year-old) implementation is a bad look.

# Wayland is more performant (as long as you don't use NVidia)

Several of the design decisions that Wayland makes are claimed in the name of performance. Namely, collapsing many layers is supposed to reduce the number of copies when moving data between different components.

However, whatever the reason, these performance gains haven't materialized, or are so anecdotal in either direction that it's difficult to claim a clear win over X11. In fact, you can find examples showing roughly a 40% slowdown when using Wayland over X11! I'm sure there are similar benchmarks claiming Wayland wins and vice versa (happy to link them as well if provided).

The problem is that, even if Wayland was twice as fast, it doesn't compare to improvements in hardware over the same period. It would've been better to just wait! The performance improvements would have to be much more substantial for this to be a reasonable argument in favor of Wayland. The fact that the question even exists as to whether one or the other is faster after a substantial period is an obvious failure.

Additionally, those performance gains don't matter if I'm not able to make use of them. For example, if I'm using the most popular graphics card vendor in my system, I should expect things to work out of the box.

# Wayland isn't just "one thing" (because Wayland doesn't do anything)

One rebuttal I've heard is that it's not an issue with Wayland, it's an issue with the compositor/extension/application. After all, "Wayland" isn't a piece of software, it's a simple protocol that other software chooses to implement!

Of course, what this means in practice is that there are multiple (usually incompatible) implementations of multiple different standards. Maybe this would be fine if the concept of a desktop operating system was completely new and unknown, but users balk when discovering things like drag and drop or screen sharing are not natively supported and are essentially still in "beta" status.

Instead of providing a better way of doing something, common features are not supported at all, and instead it's the job of everyone else in the ecosystem to agree on a standard. That's not a stunning argument in favor of replacing something that already exists and that has already been standardized in X11!

# Wayland is still under active development

Wayland has been around for only 17 years, while X11 has closer to 40 years of development behind it. Things are still under development and obviously will get better, so why complain about issues that will inevitably get fixed?

Because it's been 17 years and people are still running into major issues!

I was unpleasantly surprised when using KDE Plasma that the default display server had been changed to Wayland. I noticed very quickly on startup when I encountered enough graphical hitches to realize I was running Wayland and quickly switch back. Anecdotal experience is not enough to say this is a broad issue, but my point is that when an average user encounters graphical issues within 60 seconds of using it, maybe it's not ready to be made the default! It was only within the last 6 months that OBS stopped segfaulting when trying to launch on Wayland. I assume I'm in decent company when even the developer of a major compositor is still not able to use Wayland in 2026.

The number of "simple" utilities that seem partially supported or half-baked is incredible and seems to be a massive duplication of effort. The tooling around X11 that has been developed over the last 40 years seems to have been completely dropped and no alternative has been provided. Instead of providing an obvious transition path, Wayland has introduced even more fragmentation.

Older software that has a ton of "legacy cruft" has been tested and bugs have long since been fixed. I fully believe with another 20 years of development things will be better. The problem is that I am being forced to make the switch now. See: The push from KDE and RedHat to Wayland and dropping support for older technologies.

# Opinion of Wayland developers

This post probably best encapsulates the developer opinion towards users trying to migrate to the next iteration of the Linux desktop:

Maybe Wayland doesn’t work for your precious use-case. More likely, it does work, and you swallowed some propaganda based on an assumption which might have been correct 7 years ago. Regardless, I simply don’t give a shit about you anymore.

...

We’ve sacrificed our spare time to build this for you for free. If you turn around and harass us based on some utterly nonsensical conspiracy theories, then you’re a fucking asshole.

It's even more ironic compared to the post made a week later, expressing the same frustration with the Rust community that people have with Wayland!

Drew has since deleted this post, so I understand if he no longer stands by those opinions. However, it's a representative slice of developer sentiment towards users that are now being forced to use unfinished software. Entitlement and bullying of open-source maintainers is not appropriate, and it's understandable that the developers lash out after feeling beaten down by entitled users. However, to have some sympathy on the user side, it's likely born out of frustration of being forced to use the new hotness and then encountering breaking bugs that are impossible for the average user to work around.

It is not the fault of the original developers for building what they wanted to build. I think it's important to keep in mind that they didn't necessarily choose for Wayland to become as popular as it has or the foundation for the desktop of the future. See the diagram below:

Having Wayland as a developers-only playground is fine! Have fun building stuff! But the second actual users are forced to use it expect them to be frustrated! At this point I consider Wayland to be a fun toy built entirely to pacify developers tired of working on a finished legacy project.

# Predictions and looking forward

# Things to be excited about

Since most of this post has been overwhelmingly negative against the development of Wayland, it's instead better to learn as much as possible and look forward towards "what would I want to be able to do". Windowing technology is absolutely not "done", and instead of following other operating systems, it would be fantastic if Linux could do things no other environment could do.

For example, being able implement non-rectangular windows, exposing context actions (similar to MacOS), or making it easier to automate or script parts of the desktop environment would be incredibly exciting!

It's difficult to overstate the amount of progress in support for gaming, new (and old) hardware, as well as the amount of overall "polish". Every developer should be proud to be a part of that!

# Conclusion

After 17 years, Wayland is still not ready for prime time. Notable breakage is being documented, and adoption has been correspondingly slow.

For some users the switch is seamless. For others (including myself), they tend to bounce off after encountering workflow-breaking issues. I think it's obvious at this point that the trade-offs have not been worth the hassle.

My prediction is that within the next 5 years the following will be true:

Projects will drop Wayland support and go back to X11
There will be a new display protocol that displaces both X11 and Wayland
The new display protocol will be a drop-in replacement (similar to XWayland)
Fragmentation will still be an issue (this one's a freebie)

See you in 2030 for the year of the Linux Desktop.

# Additional reading

Included are some of the links referenced in this post as well as some additional reading.

Think twice before abandoning X11. Wayland breaks everything!
Wayland is flawed at its core and the community needs to talk about it
- https://www.reddit.com/r/linux/comments/1pxectw/wayland_is_flawed_at_its_core_and_the_community/
Going all-in on a Wayland future
- https://blogs.kde.org/2025/11/26/going-all-in-on-a-wayland-future/
Unpopular Opinion: Linux world felt stable until Wayland/GTK4 arrived
- https://www.reddit.com/r/linux/comments/1o5fmxv/unpopular_opinion_linux_world_felt_stable_until/
On Abandoning the X Server
- https://news.ycombinator.com/item?id=24920183
I'm tired of this anti-Wayland horseshit (deleted)
- http://web.archive.org/web/20211008180535/https://drewdevault.com/2021/02/02/Anti-Wayland-horseshit.html
Move fast and break things as a moral imperative (deleted)
- http://web.archive.org/web/20230605024414/https://drewdevault.com/2021/02/09/Rust-move-fast-and-break-things.html
Can I finally start using Wayland in 2026?
- https://michael.stapelberg.ch/posts/2026-01-04-wayland-sway-in-2026/
KiCad and Wayland Support
- https://www.kicad.org/blog/2025/06/KiCad-and-Wayland-Support/
Wayland Finally Gains Ground: Why 2025 is the Year of Desktop Linux Migration
- https://chandanadev.com/linux_2
What is the true adoption rate of Wayland
- https://x.com/LundukeJournal/status/1994963236827865224
Simplified history of X Hector Martin
- https://web.archive.org/web/20240110152647/https://social.treehouse.systems/@marcan/110371565062371963
Wayland breaks your bad software
- https://web.archive.org/web/20240112184154/https://oro.gay/posts/wayland-breaks-your-bad-software/
on abandoning the X server
- https://ajaxnwnk.blogspot.com/2020/10/on-abandoning-x-server.html

nanotrace: A nanosecond-scale profiler using Intel PT

2026-02-14 18:09:04 -05:00

Since this post is a bit longer, feel free to skip around or go straight to the benchmarks and gallery.

Introduction
Overview of Intel PT
Benchmarks
- magic-trace
- libxdc
Optimizing nanotrace
Applications
Final thoughts
References
Gallery

# Introduction

Computers are fast. Really fast. If you have a decently modern desktop processor running at full utilization, each core can execute >8 billion instructions per second, with each instruction taking roughly ~1/10 of a nanosecond! ^1

How do you profile or understand performance when there are that many things happening every second?

Luckily, there exists — on Intel processors, at least — a hardware feature called Intel PT that makes it possible to reconstruct the instruction stream of any running program with minimal overhead, generally 2-20% of program runtime. It even includes accurate cycle and timing information!

Unfortunately, the format is extremely compressed and traces can be very large, up to ~4GB/s. Additionally, getting an accurate view of execution requires decoding each individual instruction, which for x86 can be challenging, as I mentioned in a previous post. This, along with tooling that has been historically lacking, means Intel PT has languished as a potentially extremely useful, but underused, feature.

In this post, I describe a PT decoder I recently developed called nanotrace that beats the state-of-the-art in this area, and gives greater insight into what a program is doing, down to individual instructions.

All the traces shown below were generated using nanotrace and displayed using the excellent Perfetto.

Before this post gets too long, I wanted to show a quick taste of what it can do. Here's a trace from running python3 -c 'print(1+2)'. You can see all the calls made by the print (builtin_print.lto_priv.0) as well as how long it took (~11us). Take a look at all the things print does, then try zooming out!

# Overview of Intel PT

This is an introduction to the actual format used by Intel PT, if you're already familiar or want to skip ahead, I'd recommend looking at the benchmarks.

The core problem when generating an instruction trace is giving enough information to decode the trace without overwhelming the available memory bandwidth of the system. To that end, Intel PT works by generating the minimum amount of data required to reconstruct program flow. This means traces are comprised almost entirely of the following:

Taken/Not-Taken (TNT) packets, which contain individual bits for each conditional taken or not taken
Target IP (TIP) packets, for updating the IP when it cannot be directly inferred (for example with indirect jumps or returns)
Flow update (FUP) packets, for when instruction flow is completely interrupted (for example when receiving an interrupt or making a syscall)

There's some additional miscellaneous packets that are used for indicating system frequency and time (CBR, MTC and TSC packets), as well as an accurate cycle counter (CYC), if enabled. On newer processors there's support for writing custom data directly to the trace (PTWRITE). With precise event-based sampling, you can even do fancy things like dumping the full register state when specific PMUs overflow. Unfortunately, on my processor (Kaby Lake) I don't have access to most of those, so we'll keep the discussion to just the instruction trace for now.

You can see what packets are emitted with the following commands:

$ perf record -e intel_pt//u -- sleep 1
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.031 MB perf.data ]

$ perf report --dump-raw-trace
// ... <snip> ...

0 0 0x960 [0x30]: PERF_RECORD_AUXTRACE size: 0x6880  offset: 0  ref: ...  idx: 7  tid: ...  cpu: 7
.
. ... Intel Processor Trace data: size 26752 bytes
.  00000000:  02 82 02 82 02 82 02 82 02 82 02 82 02 82 02 82 PSB
.  00000010:  00 00 00 00 00 00                               PAD
.  00000016:  19 22 8d fb ba d7 80 0c                         TSC 0xc80d7bafb8d22
.  0000001e:  00 00 00 00 00 00 00 00                         PAD
.  00000026:  02 73 c2 78 00 84 00 00                         TMA CTC 0x78c2 FC 0x84
.  0000002e:  00 00                                           PAD
.  00000030:  02 03 2c 00                                     CBR 0x2c
.  00000034:  02 23                                           PSBEND
// ... <snip> ...
.  000002ba:  31 c7 47                                        TIP.PGE 0x47c7
.  000002bd:  a2                                              TNT NTNNNT (6)
.  000002be:  0e 00                                           TNT TT (2)
.  000002c0:  59 37                                           MTC 0x37
.  000002c2:  94                                              TNT NNTNTN (6)
.  000002c3:  d4                                              TNT TNTNTN (6)
.  000002c4:  d0                                              TNT TNTNNN (6)
.  000002c5:  80                                              TNT NNNNNN (6)
.  000002c6:  c0                                              TNT TNNNNN (6)
.  000002c7:  84                                              TNT NNNNTN (6)
.  000002c8:  da                                              TNT TNTTNT (6)
.  000002c9:  c4                                              TNT TNNNTN (6)
.  000002ca:  98                                              TNT NNTTNN (6)
.  000002cb:  80                                              TNT NNNNNN (6)
.  000002cc:  12 00 00                                        TNT NNT (3)
.  000002cf:  7d 17 4a 16 0e 6f 7f 00                         FUP 0x7f6f0e164a17
.  000002d7:  00                                              PAD
.  000002d8:  01                                              TIP.PGD no ip
// ... <snip> ...

You can also get the full instruction trace from perf with the following commands:

$ perf script --insn-trace
/// ...
   hello 3214753 [003] 1.702313143: 401139 main+0x0 (./hello)     insn: 48 83 ec 08
   hello 3214753 [003] 1.702313143: 40113d main+0x4 (./hello)     insn: 48 8d 3d c0 0e 00 00
   hello 3214753 [003] 1.702313143: 401144 main+0xb (./hello)     insn: e8 e7 fe ff ff
   hello 3214753 [003] 1.702313143: 401030 puts@plt+0x0 (./hello) insn: ff 25 ca 2f 00 00
   hello 3214753 [003] 1.702313143: 401036 puts@plt+0x6 (./hello) insn: 68 00 00 00 00
   hello 3214753 [003] 1.702313143: 40103b puts@plt+0xb (./hello) insn: e9 e0 ff ff ff

If you have libxed installed, perf can use it to decode instructions:

$ perf script --insn-trace --xed
// ...
   hello 3214753 [003] 1.702313143: 401139 main+0x0 (./hello)     sub rsp, 0x8
   hello 3214753 [003] 1.702313143: 40113d main+0x4 (./hello)     lea rdi, ptr [rip+0xec0]
   hello 3214753 [003] 1.702313143: 401144 main+0xb (./hello)     call 0x401030
   hello 3214753 [003] 1.702313143: 401030 puts@plt+0x0 (./hello) jmp qword ptr [rip+0x2fca]
   hello 3214753 [003] 1.702313143: 401036 puts@plt+0x6 (./hello) push 0x0
   hello 3214753 [003] 1.702313143: 40103b puts@plt+0xb (./hello) jmp 0x401020

For more information about the actual trace information, I'd recommend taking a look at the Intel Architecture Manual. Relevant information is in Volume III, Chapter 36 at the time of this writing.

The main thing you'll notice from the raw output is that there's very little instruction information! We have to decode all of that ourselves, which means disassembling each instruction until we hit a branch, then updating the IP based on a single bit of output. As we'll see below, roughly 1 in 20 instructions are branches. So, if we have to decode ~20 instructions for each bit of output, that will slow down decoding considerably! Luckily, there's a couple tricks we can use to speed up decoding, based on some things we know about how most programs execute.

But first, let's compare to some existing tools to see how they perform.

# Benchmarks

I've tried to keep the following benchmarks as fair as possible. If there's additional tools that you'd like to see compared, please let me know. I chose tools from both the profiling and fuzzing domains to show that we perform well in both, and I expect nanotrace would perform will if adapted to other applications as well.

# `magic-trace`

magic-trace is an awesome tool that uses Intel PT under the hood to generate a function-level of every function executed in a binary. Because of the incurred overhead when decoding, it generally is only used to capture the last 10ms or so of a program's execution. Since I was having trouble running magic-trace decode on a previously recorded session, I've instead benchmarked magic-trace run and included the overhead of running perf when comparing against nanotrace.

Here, we're demonstrating both magic-trace and nanotrace generating a trace.fxt.gz file for use with Perfetto, with nanotrace producing the same result roughly 8-10x faster than magic-trace. As an aside, I was having trouble getting the debuginfod working with magic-trace, so I would argue our trace is also significantly more useful since it is able to resolve functions that would otherwise show up as [unknown].

# `libxdc`

One of the most promising uses of Intel PT is for fuzzing. libxdc aims to be the best at providing a bitmap from a decoded trace for use with a fuzzer like kAFL. It claims to be the fastest for repeated runs of similar programs — like one would expect from a fuzzing campaign — and it aggressively caches results for accelerating future runs.

Here, we're running against libxdc with the same input data and generating an identical output bitmap across 5 trials. The only difference between the two programs is the decoder implementation.

As shown above, we are roughly 1.4-1.6x faster than libxdc while providing an identical bitmap. To put that in perspective, the reference implementation from Intel is another 30-40x slower than that!

# Optimizing `nanotrace`

From the above, we can see a significant performance difference from other tools. How is nanotrace able to outperform other implementations?

# Caching block transitions

It's a common myth that 80% of the running time is spent in 20% of the code. While that generally isn't true across the execution of an entire program, it absolutely is true for local segments, which we can use for our own benefit.

To demonstrate the sort of behavior we often see when decoding traces, I ran an instruction trace on the Python example shown above (python3 -c 'print(1+2)'). From that trace, here are the 20 most commonly executed regions of code:

 226939      7f8c4baf36ae               ret 
 226939      7f8c4baf36aa               add rax, qword ptr [rdi+0x8]
 226939      7f8c4baf36a8               jz 0x7f8c4baf36af
 226939      7f8c4baf36a4               cmp rax, 0xffffffffffffffff
 226939      7f8c4baf36a0               mov rax, qword ptr [rdx+rax*1]
 226939      7f8c4baf369c               shl rax, 0x4
 226939      7f8c4baf3699               mov rax, qword ptr [rdi]
 226939      7f8c4baf3697               jnz 0x7f8c4baf36af
 226939      7f8c4baf3694               cmp qword ptr [rdx], rax
 226939      7f8c4baf368d               mov rax, qword ptr [rip+0x1f15c]
 226939      7f8c4baf3684               mov rdx, qword ptr fs:[0x8]
 226939      7f8c4baf3680               nop edx, edi
 141282      7f8c4b57b530               jnz 0x7f8c4b57b520
 141282      7f8c4b57b52d               cmp r15, rdx
 139169      7f8c4b57b52b               jz 0x7f8c4b57b550
 139169      7f8c4b57b528               test rdx, rdx
 139169      7f8c4b57b524               add rax, 0x38
 139169      7f8c4b57b520               mov rdx, qword ptr [rax+0x68]
  83839      7f8c4b62a32f               jnz 0x7f8c4b62a338
  83839      7f8c4b62a328               test byte ptr [rax+0xa9], 0x40

From the above, we can see that ~20% of the most executed locations are branches (and closer to 30% if you include ret).

However, you might notice I'm still including the IP, and not looking at unique instructions. Here's that comparison, showing that the total number of conditional jumps is actually closer to 2% across the entire program:

1012712 endbr64
 895201 ret
 519760 test rax, rax
 486278 test eax, eax
 474129 push rbp
 474129 mov rbp, rsp
 357972 push rbx
 342694 test rdi, rdi
 340328 xor eax, eax
 333077 test rdx, rdx
 315588 shl rax, 4
 309239 pop rbp
 302731 pop rbx
 294050 push r12
 292344 pop r12
 258789 cmp rax, -1
 247148 push r13
 247141 pop r13
 245254 mov rax, qword ptr [rdi]
 239582 je 7

To elaborate on the above, here's a larger table showing types of instructions and their frequency. You'll notice it forms a Pareto distribution, where the most common instructions are executed almost 1,000,000x more often than the least executed, which often run only once.

From the above, we see that the most commonly executed regions are blocks containing almost 20% conditional jumps. We also see that executed conditional jumps across the entire program are less than 2%. How does that make sense?

The answer is that conditional jumps are sparse, but are generally executed many times in "hot loops". For example, when iterating over items in an array or linked list.

Therefore, caching information about every conditional jump we see is likely not to be prohibitively expensive, but will yield significant performance improvements for the cases where we need to decode that branch more than once. As we've seen above, this the most common case for conditional jumps. What this means is that a branch that was executed previously is likely to be executed again, so we can cache any interesting events that may have happened from decoding that branch and replay them later.

# Custom x86 disassembler

Since a significant amount of time is spent decoding instructions, it's important that it's fast. As mentioned in a previous post, instruction decoding is handled by a very tight decode loop with a specialized instruction table that only contains information needed for decoding a trace. Assuming everything was the same between two implementations, the disassembler alone will provide 1.5x lift when decoding instructions.

Since I've already written about the disassembler, I won't write much more about that here — except to refer you to the above post.

# Keep it simple, stupid

After the above optimizations, I started looking further afield for more clever approaches to increasing performance, including a custom JIT to build a "meta-program" to accelerate decoding in specific parts of a program. What I realized is that it's much more difficult to do better at optimization than the hardware itself. In other words: more clever optimizations make it harder for the processor to understand and run your code, so keep it simple!

We do have numerous tiny optimizations, including how we lookup instructions as part of decoding, but they do not contribute nearly as much to the total speed than the above two. Including additional optimizations like PGO+LTO I would expect another 5-10% speedup without changing the current implementation.

# Applications

# Profiling

By far the most common and known application of instruction-level tracing is for profiling. nanotrace makes this application more useful since it reduces time to action for a recorded trace. For visualization, a difference in 10x is the difference between taking a coffee break and a full lunch. Having cycle information also adds another way of identifying performance bottlenecks. By comparing the reported cycle time with the sum of the expected latency of each instruction, it's possible to compare the actual versus expected time to execute each instruction. This can be useful to identify if, for example, memory loads in certain places are taking much longer than expected. perf can somewhat do this by calculating instructions-per-cycle (IPC) using cycle information, but using perf report generally has all tail-latencies averaged together so it becomes much less useful.

# Fuzzing

For the libxdc comparison, a 1.4x uplift isn't as drastic an improvement, but it proves there's still more juice left to squeeze when it comes to this technology. Additionally, a feature that I haven't mentioned is that it's possible to filter IP ranges using either built-in processor features or build it into the decoder itself. This means you can restrict tracing to specific areas of interest and makes it possible to have an arbitrarily complicated harness that instruments a target without polluting the bitmap with irrelevant information. Another case might be when you're only interested in fuzzing or profiling generated code, as might be the case when writing a JIT. You could even support Intel PT on an unsupported architecture by simply recording it running inside QEMU! Unfortunately, I wasn't able to get perf inject to work when recording with -e intel_pt//u, so that'll have to wait for another time. Stay tuned!

# Debugging

One last application that's particularly interesting to me is debugging. Having a perfectly accurate instruction trace means it's possible to do things like reverse debugging, where each breakpoint and watchpoint can be tracked backwards by checking the last time a set of instructions was hit. rr is a project that does something similar, but instead records snapshots so it can re-execute parts of the program on the fly.

# Final thoughts

Currently this tool is not open-source, but if you're intested in how it could be useful for you or your business, please feel free to drop me a line!

I really wish every system I interacted with had something like Intel PT (looking at you, AMD!). Although for now, as mentioned above, it is still possible to run through emulation and get trace information with minimal overhead (although that still comes with significant slowdown from emulation). I'd be interested in some future Bizarro-Rosetta that is closer in parity to performance.

With numbers coming out from the newly released Panther Lake, I'm hoping the market share of devices with this feature will increase.

I expect to write more about some of the applications developed using nanotrace, as well as any performance issues uncovered from running on open-source tools. A profiler is only as good as the performance issues it uncovers, so more to come on that front.

# References

Here's some additional resources if you're curious in performance tracing or other tools like this:

All my favorite tracing tools
- Article by Tristan Hume that covers magic-trace, as well as instrumentation-based approaches. Covers many other areas of performance engineering and is worth the read.
Enhance performance analysis with Intel Processor Trace
- Highly recommended series for learning more about how to use Intel PT with existing tools.
Magic-trace: Diagnosing tricky performance issues easily with Intel Processor Trace
- Write-up from Jane Street on identifying their performance issues at scale.
Reverse Engineering and Control-Flow Analysis with Intel Processor Trace
- A recent article that describes using Intel PT for understanding control flow inside of an obfuscated binary. Very cool application of Intel PT
Un-bee-lievable Performance
- Article from Trail of Bits where they add a pre-caching step to accelerate decoding that demonstrates speed-ups for some applications.
Cheat sheet for Intel Processor Trace with Linux perf and gdb
- Quick reference for using Intel PT and adding to your workflow if using gdb or perf.

# Gallery

I gathered a couple extra traces as I was developing nanotrace that I wanted to share, including one to compare the overhead of dynamic versus static linking.

For hello.c, I'm compiling a basic hello world example:

#include 

int main(int argc, char **argv) {
  printf("Hello, world!\n");
  return 0;
}

# ./hello (dynamically linked)

# ./hello (statically linked)

# lua -e print(1+2)

# cc1 hello.c

# clang hello.c

With a 4GHz 4-core processor, a single cycle takes 0.25ns. Within each cycle, a single core can process ~1-4 instructions (depending on code). Assuming no stalls, etc. this means, across 4 cores, each second your computer (at full utilization) can execute ~32 billion instructions.

Instruction decoding at the speed of light

2026-01-04 12:57:58 +00:00

As part of another project, I found myself needing an x86 disassembler. It is not important that it is detailed in its output, but it is important that it is fairly fast and mostly complete. I wrote my own that outperforms the alternatives for my particular use case, and thought it was an illustrative example of the sort of assumptions people make when it comes to performance.

More generally, what is the speed limit of this operation, and how do we define "fast"?

# The problem with x86

For my problem, I need to take a stream of bytes and count the number of valid instructions. Because instructions are consecutive, we need to know the length of each one before moving onto the next, therefore making decoding inherently serial.

Every architecture has a well-defined set of available instructions defined as the instruction set architecture (ISA). Most architectures have constant-length instructions (generally 2-4 bytes) and finding the next instruction is a fairly simple operation:

Is this instruction 2 or 4 bytes?
Is this instruction a jump?
If it is a jump, decode jump target and GOTO 1
If not a jump, increment counter by 2 (or 4) bytes
GOTO 1

The x86 architecture (or x86_64/amd64 if you're targeting something made after 2003) is somewhat unique in that each instruction can be anywhere from 1-15 bytes. This is because, in order to maintain backwards compatability, every new generation of Intel and AMD processors finds space for new (usually longer) instructions in the gaps left over from previous versions of the ISA. This dates back to the instructions implemented on the original 8080, an 8-bit microprocessor originally released in 1974.

The original ISA has expanded several times over the last five decades to support extended registers, larger embedded constants, SIMD, cryptographic extensions, etc. Each instruction is made up of roughly three parts:

Zero or more modifiers or prefixes (REX, LOCK, etc.)
The actual opcode (JMP, MOV, etc.)
Zero or more operands (register, immediate, memory)

Each of the above components can be one or more bytes. Consider the following operation that moves the constant 0x1337 into register rax:

 48 | c7 | c0  | 37 13 00 00    mov rax, 0x1337
REX | OP | REG | IMM

In this example:

The prefix is 0x48 (use the 64-bit rax instead of the 32-bit eax)
The opcode is 0xc7 (mov)
The operands are:
- 0xc0 - destination is register 0 (rax)
- 0x37130000 - little-endian 4-byte constant (0x00001337)

The above explanation is oversimplified for the purposes of illustration since the actual decoding is more complex still, but serves to demonstrate that the actual decoding operation is not as simple as checking the first N bytes and exiting with a known opcode, operands, and instruction length.

A more realistic example is something you'd see in objdump, for example:

0000000000001180 <main>:
    1180:	41 57                	push   r15
    1182:	41 56                	push   r14
    1184:	53                   	push   rbx
    1185:	be 00 ca 9a 3b       	mov    esi,0x3b9aca00
    118a:	31 ff                	xor    edi,edi
    118c:	ba 07 00 00 00       	mov    edx,0x7
    1191:	b9 22 00 00 00       	mov    ecx,0x22
    1196:	41 b8 ff ff ff ff    	mov    r8d,0xffffffff
    119c:	45 31 c9             	xor    r9d,r9d
    119f:	e8 9c fe ff ff       	call   1040 <mmap@plt>
    11a4:	48 89 c3             	mov    rbx,rax
    11a7:	ba ff c9 9a 3b       	mov    edx,0x3b9ac9ff
    11ac:	48 89 c7             	mov    rdi,rax
    11af:	be 90 00 00 00       	mov    esi,0x90
    11b4:	e8 a7 fe ff ff       	call   1060 <memset@plt>
    11b9:	c6 83 ff c9 9a 3b c3 	mov    BYTE PTR [rbx+0x3b9ac9ff],0xc3
    11c0:	e8 6b fe ff ff       	call   1030 <clock@plt>
    11c5:	49 89 c6             	mov    r14,rax
    11c8:	31 ff                	xor    edi,edi
    11ca:	ff d3                	call   rbx
    11cc:	49 89 c7             	mov    r15,rax
    11cf:	e8 5c fe ff ff       	call   1030 <clock@plt>
    11d4:	4c 29 f0             	sub    rax,r14
    11d7:	f2 48 0f 2a c0       	cvtsi2sd xmm0,rax

There are additional complexities since x86 also supports several different modes of operation for compatability with older systems, which can change how different instructions get decoded.

Those are mostly irrelevant for my problem. For simplicity, assume for the rest of this article we are only considering 64-bit x86.

# Bounding compute and IO

So, given the above constraints, what is theoretically the fastest implementation we could hope to achieve?

To help level-set on "what is fast", it's illustrative to see both: how fast are we able to read data from memory, and how fast the actual hardware is able to decode (and execute!) the equivalent instructions we're trying to disassemble.

Here's a simple test harness that allocates and executes 1 GB of memory filled with NOPs. The main goal of this benchmark is to understand how fast instruction decoding is in hardware, and the maximum throughput of a decoder if we were truly IO-bound.

#include "stdint.h"
#include "stdio.h"
#include "string.h"
#include "sys/mman.h"
#include "time.h"

#define KILOBYTES 1000ull
#define MEGABYTES 1000ull * KILOBYTES
#define GIGABYTES 1000ull * MEGABYTES

int main(int argc, char **argv) {
  clock_t start_time, end_time;
  uint64_t size = 1 * GIGABYTES;
  uint8_t *code = mmap(NULL, size, PROT_READ | PROT_WRITE | PROT_EXEC,
                       MAP_ANONYMOUS | MAP_PRIVATE, -1, 0);

  // fill with NOPs
  memset(code, 0x90, size);
  code[size - 1] = 0xc3; // ret

  start_time = clock();
  // execute 1 billion NOPs, then return
  uint8_t val = ((uint64_t (*)(uint64_t))code)(0);
  end_time = clock();

  printf("execute %lu bytes in %.2f ms (val: %.2x)\n", size,
         (double)(end_time - start_time) * 1000.0 / CLOCKS_PER_SEC, val);

  // MAP_POPULATE will pre-fault all the memory in this range so we aren't
  // including the cost of page-faults in our benchmark
  uint8_t *output = mmap(NULL, size, PROT_READ | PROT_WRITE,
                         MAP_ANONYMOUS | MAP_PRIVATE | MAP_POPULATE, -1, 0);

  start_time = clock();
  memcpy(output, code, size);
  end_time = clock();

  // Use the last value of output so the memcpy isn't optimized out
  printf("memcpy %lu bytes in %.2f ms (val: %.2x)\n", size,
         (double)(end_time - start_time) * 1000.0 / CLOCKS_PER_SEC,
         output[size - 1]);

  return 0;
}

On my machine (Intel i7-7700K), I get roughly 70ms for instruction execute, and 55ms for memcpy, which corresponds to a total throughput of 14.29GB/s and 18.18GB/s, respectively. If we refer to the expected throughput for Kaby Lake, we can see that one NOP should take roughly 0.25 cycles, meaning we should be able to execute 4 NOPs per cycle. At the base clock of 4.2 GHz, we would expect to be able to execute ~16.8 billion NOPs per second. Or, if measuring in bytes, a total throughput of 16.8GB/s. Since the machine I'm running on has other processes running, I expect some loss from other tasks pulling cycles.

The memcpy benchmark is a bit faster. Testing memory bandwidth with other benchmarks, I'm able to pull up to ~22GB/s on a single core, which I can accept when compared to similar comparisons. For us, this represents the maximum throughput when doing as little work as possible on each byte, and is what we'll compare to when benchmarking implementations in software.

# So, how does it perform in software?

I'll be using disas-bench for these comparisons. The benchmark builds and tests all major disassemblers for x86 and puts performance in the same units we have above. How do they all perform?

We can see the fastest is yaxpeax at ~270MB/s, roughly 80x slower than memcpy, or almost 50x slower than executing those instructions natively.

The slowest (decode-only) benchmark is Zydis, a full 10x slower than yaxpeax at around 30MB/s.

So, how would we make it faster?

# Back to basics

Let's go back to our initial benchmark and start with the simplest implementation: a "decoder" that checks if an instruction is a NOP, and returns 1 if so. Every other instruction will be considered invalid:

static uint8_t insn_sz(const uint8_t *insn) {
  if (insn[0] == 0x90) // NOP
    return 1;
  return 0;
}

  // in main...
  start_time = clock();
  uint64_t tot_insns = 0;
  for (uint8_t *insn = code; insn < code + size;) {
    uint8_t sz = insn_sz(insn);
    if (sz == 0) // invalid
      break;
    insn += sz;
    tot_insns += 1;
  }
  end_time = clock();

  printf("decode %lu bytes in %.2f ms (use var: %lu)\n", size,
         (double)(end_time - start_time) * 1000.0 / CLOCKS_PER_SEC, tot_insns);

  // ...

This implementation completes in 310ms, marking a total throughput of 3.23GB/s. Putting those numbers in perspective with the above benchmarks:

Pretty good, but it doesn't do much of anything! So what if we just keep adding instructions? We can add a specific check for each instruction we encounter and return its length:

static uint8_t insn_sz(const uint8_t *insn) {
  if (insn[0] == 0x90) return 1; // NOP
  else if (insn[0] == 0x48 && insn[1] == 0x01 && insn[2] == 0x00) return 3;
  else if (insn[0] == 0x48 && insn[1] == 0x01 && insn[2] == 0x01) return 3;
  else if (insn[0] == 0x48 && insn[1] == 0x01 && insn[2] == 0x02) return 3;
  else if (insn[0] == 0x48 && insn[1] == 0x01 && insn[2] == 0x03) return 3;
  else if (insn[0] == 0x48 && insn[1] == 0x01 && insn[2] == 0x05) return 7;
  else if (insn[0] == 0x48 && insn[1] == 0x01 && insn[2] == 0x06) return 3;
  else if (insn[0] == 0x48 && insn[1] == 0x01 && insn[2] == 0x07) return 3;
  else if (insn[0] == 0x48 && insn[1] == 0x01 && insn[2] == 0x08) return 3;
  else if (insn[0] == 0x48 && insn[1] == 0x01 && insn[2] == 0x09) return 3;
  // ~1 million lines follow
  // ...
  return 0;
}

Besides getting a bit unwieldy, we start running into other issues:

$ clang bench.c -O3 -o bench
clang: error: unable to execute command: Segmentation fault (core dumped)
clang: error: clang frontend command failed due to signal (use -v to see invocation)

We can reduce the number of conditionals by doing an early out, but we'll still start running into segfaults. Most compilers aren't designed for compiling gigabytes of code.

Looks like we'll need something better.

# Something (a bit) better

Looking at the problem more generally, the problem we're trying to solve is the equivalent of a trie lookup. Each byte of an instruction represents a possible branch point until we reach a valid instruction. Here's a more visual way to see the actual traversal:

Instead of a series of conditionals, we could've written the above as something like:

switch(insn[0]) {
  case 0x90: return 1;
  case 0x48:
    switch(insn[1]) {
      case 0x90: return 2;
      // ...
    }
  // ...
}

Where each switch represents a traversal down one node of the trie. Doing this for the rest of our instructions we get ~400MB/s, placing it above yaxpeax:

One last thing we can do is convert to a table based lookup. Because our switch-case implementation executes mostly conditionals when decoding NOPs, we expect to be able to perform better when execution flow is more consistent.

Each node in our trie is represented with a static array of 256 entries. The core decoder loop also ends up being a bit smaller.

struct table_entry;

typedef struct table_entry {
  struct table_entry* insn_tbl[0x100];
  uint8_t insn_sz[0x100];
} table_entry;

static table_entry insn_sz_ = {
    .insn_sz = {0, 0, 0, 0, 2, 5, 0, /* ... */ },
    .insn_tbl = {&insn_sz_00, &insn_sz_01, /* ... */ },
};

// ...

uint8_t insn_sz(uint8_t *insn) {
  table_entry *cur_table = &insn_sz_;
  for (uint8_t i = 0; i < 10; ++i) {
    if (cur_table->insn_tbl[insn[i]] == NULL)
      return cur_table->insn_sz[insn[i]];
    cur_table = cur_table->insn_tbl[insn[i]];
  }
  return 0;
}

This implementation ends up at around ~520MB/s, placing it above our previous switch-case implementation and outperforming the next best by around 1.9x.

Can we make it go faster? Without repurposing parts of the x86 decode hardware, I don't see an obvious way to do so in software, so we'll end it there for today. As a reminder, here's our final round-up with the speed of native decode in comparison to the other entries in our list:

Hardware is really fast!

# Conclusion

The final implementation is simple, and does little besides give enough information to decode the next instruction — I have a similar decoder specifically for filtering the type of instruction (jumps, moves, etc.). Together those are enough for my use case and outperform the alternative by roughly 2x.

However, not every task warrants this level of performance engineering, so what's the point? My hope is not that you apply this level of effort to every problem, but at a high level take away some sense of just how fast your computer is and to have a rough metric of how long things should take.

As a final thought, this post is an exercise in working backwards from the speed of light. Working from basic principles, what is the simplest, fastest example of something that works, and how can we make it more complete without sacrificing performance? Additionally, what is the core problem we are trying to solve? Being able to generalize the problem can lead to a better approach pulled from further afield.

If you have other thoughts, please let me know!

# Some remaining questions

There are a couple questions that came up while working through this that I'll answer here:

> Is this why ARM is more efficient than x86?

No. It doesn't help, but the complexity of modern processors is staggering and attributing "poor efficiency" to any one factor is wrong at best and misleading at worst. Something like the above lookup structure is largely stamped out in hardware where circuitry is cheap. There are lots of neat tricks for parallelizing decode in hardware but it's largely a solved problem.

> It seems like some instructions are super common, what if you put in a special case for those?

This actually makes things slower for microbenchmarks and ends up being irrelevant on actual binaries. This is because the benefit from special casing the most common instructions carries a high mispredict penalty when decoding other instructions.

> What about SIMD?

Unless you're on an Intel Xeon or something this won't help much. The byte-to-byte dependency chain is very tight, and there isn't much room to squeeze out benefits from parallel work. Decoding multiple streams would be faster if non-consecutive SIMD loads were faster, which on my machine they are not.

> What about Rust?

The main advantage of re-writing everything in Rust would be placing everything in a single compilation unit. Unless there are aggressive opportunities for inlining, I do not see a benefit to doing so. I would, however, love to be shown wrong.

My website looks better than yours

2025-11-23 19:09:47 +00:00

Hey, why does every programmer's website look like this?

# Aesthetics are an anti-signal

The purpose of good design should be to whet the appetite, not to elevate a bad product. The best programmer in the world is unlikely to be the best writer or the best designer, or even a half-way decent one; so much of what they produce likely does not reach their audience. Instead, it gets displaced by something that was made by a better designer, who is vice-versa likely not the best programmer nor the best writer.

This has led to a distrust of anything that has a "sheen" to it. I've been burned too many times from trying a product that presents itself well and then fails when used in production. It's implicitly asking: "Are you spending more time making it look good than making it work well?"

Especially compared to the success of blogs like danluu.com, it seems clear that something does not need to look good to succeed, so why should I be bothered?

I would argue for most people it's a barrier they have to look past, rather than something that specifically brings them joy to engage with. There's plenty of good work that people will overlook purely because of how it's presented. Dan's success is a testament to the quality of his writing that most people would not be able to overcome from presenting their work in the same way.

And programmers care about aesthetics as much as anyone! Take a look at r/unixporn and you'll see stuff like this:

There's an inherent joy in seeing something nice, that makes people want to engage with it. So why is it so difficult to make something that looks good, especially when programmers obviously do have taste? It seems like it's easier to make a good looking operating system than a good looking website!

# Hey slopbot, make me a website

It's now much easier to make something that looks decent; so why was it so difficult to do before?

In part it's due to the historical development of the modern web, and decisions made since then that make it difficult to make something that appeals to modern taste. The amount of inertia behind the modern web is absolutely staggering, so it seems unlikely that something will take its place anytime soon.

So whatever, the amount of boilerplate required to make something half-way attractive is out of the hands of most people (see: every WYSIWYG website builder). Modern software tools have failed the user if the only way to use them productively is through the use of an LLM. Non-software developers seem to be able to use Microsoft Excel just fine! Turns out we just needed fancy auto-complete.

# LLMs are floor-raising, not bar-raising

"AI" can make anyone average, which is great if you're below average. I'm perfectly happy to admit I am a below average web developer (as demonstrated by the previous design of this site!). It would be great if I were better at it, and I want to be an expert in as many areas as I can, but that is not my current goal. The purpose of this website is to be a place to demonstrate my ideas and work in a compelling way.

Suddenly, I can be better than 50% of people? Compared to most websites, I'll happily take that. For the re-design of this website, I used an LLM for a proof of the initial design, then customized to my needs and integrated with the backend codebase myself.

So don't take this website as an example of "good design". I consider it to be a representation of how far ~8 hours of effort will get the average developer in 2025.

# I wish it was as easy to make a website as a TUI

Whatever tools you give someone, if the need is great enough, they will figure out how to turn it into what they want. I've watched a non-programmer build a tag-based file structure using MacOS hardlinks. It is incredibly impressive to see and works well for them, but is that truly the best the software world has to offer?

Maybe there are better tools for file organization, but for creating websites? Apparently not, since the most aesthetically-pleasing thing most programmers can produce in 8 hours is a terminal application (a platform that saw its heyday in 1985).

# In closing

Programmers do care about aesthetics. However, we've made it too hard on ourselves to make something that looks good, and the people with a sense of taste use it to elevate things that don't work well.

LLMs can elevate your design if it is below average, and likewise will make your writing worse if it is above average. If you have work that you want to present and just need a place to showcase it, it is easier than ever to make a place for it. The average is now the bare-minimum.

Stepping away from open source

2020-08-01 20:34:08 +00:00

Starting September 1st, I'm shutting down the user-facing portion of invidio.us and stepping away from development. If you have an account on the site, you can export your data here, which will allow you to import everything to another instance or self-host.

The API will continue to function until October 1st, to give time for any services relying on it to migrate to other solutions. If you're a patron on Liberapay, I would urge you to switch your donation to iv-org, as that will ensure it goes to the stewards of the project going forward.

At the time of this writing, the main Invidious instance has roughly 35,000 users. It's difficult for me to even begin to think of those as actual people, and not just a number. At that level, the amount of feedback I receive, which I would consider rather small in comparison to others I've seen, is overwhelming. It's demoralizing not being able to respond to all of it properly. Even more so when I've received something that deserves an involved response, but I haven't had the time to reply, which counter-intuitively means the best comments are least likely to receive any response. To echo antirez: it's crushing not being able to be "there" for people.

Lately, I've felt myself withdraw much more in return. For any project though, especially something that requires maintanence, it's difficult to actually "unplug" and mentally recharge. Turning off notifications has helped, but it's definitely taken its toll.

For any project of a certain size, it's more important for the lead to be a manager more than a developer. As an example, it's mostly well-known that Linus Torvalds hardly does any coding anymore. At that point working on a side-project becomes a job, and not one I particularly enjoy. If I had known that I would've ended up in that position on a passion project, I'm curious if I would have made it open-source, or started working on it at all.

In either case, I consider myself extremely lucky to have people that can manage the project in future, and can only imagine what it would be like for someone without that luxury. I can't help shake the feeling that somewhere, the software I use is being developed solely by volunteers who would rather quit, but don't have the ability to say "no". This post has been delayed for that very reason, especially when so many people have given so much, and I feel they deserve a proper goodbye. At this point though, I can't find the strength to leave on better terms.

I don't want this to read like I somehow feel underappreciated or underpaid. I'm not motivated by praise or money. After working on the same thing for three years, I'm just tired and ready to move on.

An introduction to syntax.cr

2019-03-19 12:56:57 -05:00

# Introduction

The .tmLanguage format is currently the most popular tool for adding syntax highlighting to popular editors. It is supported by Github, VSCode, Atom, Vim, TextMate (after which it is named), and many others.

However, it is extremely limited in its ability to actually understand the language the user hopes to use (packages are often limited to naive keyword highlighting) which often leads to a misleading or unexpected result. In order for an editor to provide useful information to the user, it is necessary for the editor to properly parse the given input.

With that in mind, here is a short introduction to syntax.cr.

# Description of a language

People familiar with the original Perl library's Scanless Interface can skip this section, and go straight to colorizing the given grammar.

The language used to describe grammars is an extended form of EBNF. Each rule in a language can be described as:

S ::= a b c

Sequences (highlighted in blue) can be expressed in the form:

# Zero or more
S ::= a*

# One or more
S ::= a+

And alternations (highlighted in green) can be expressed as:

S ::= a | b

Each rule can have associated adverbs (highlighted in cadetblue), which provide extra meaning that cannot be captured in pure EBNF. For example:

S ::= a* separator => comma

This specifices a sequence of 0 or more a separated by commas.

At this point it is helpful to provide a simple grammar as an example. Here is one of JSON:

# Adapted from https://gist.github.com/pstuifzand/4447349
:start ::= json
json ::= object | array

object ::= '{' members '}'
members ::= pair* proper => 1 separator => comma
pair ::= string ':' value

value ::= string
  | object
  | number
  | array
  | true
  | false
  | null

array ::= '[' elements ']'
elements ::= value* proper => 1 separator => comma

comma ~ ','

string ::= '"' in_string '"'
in_string ~ /([^"\\]|\\[\"\\\/bftnrt]|\\u[a-fA-F0-9]{4})*/
number ~ /-?([\d]+)(\.\d+)?([eE][+-]?\d+)?/

true ~ 'true'
false ~ 'false'
null ~ 'null'

:discard ~ whitespace
whitespace ~ [\s]+

There is a clear distinction between ~ and ::=. Rules that have a ~ are called L0 rules. They define tokens, and can not have adverbs assigned directly to them. Rules that have a ::= are called G1 rules and define the structure of the language. For more information see the Marpa FAQ.

The proper adverb specifies whether or not there can be a trailing separator.

The rest of the above constructs should be fairly self-explanatory, however I would recommend playing with the demo to better explore how it works. You can edit the grammar on the left and the input on the right to colorize your given language.

To implement precedence and associativity (for example in a calculator), the || operator can be used in rules, like so:

Expression ::= Number
  | Expression '*' Expression
 || Expression '+' Expression

The documentation on metacpan.org provides more information.

# Colorizing a grammar

So now we have a grammar described in EBNF, how do we use that to highlight our language?

Each rule can be tagged with a color and/or bgcolor that corresponds to CSS's color and background-color attributes, respectively. An example rule looks like:

object ::= '{' members '}' bgcolor => blue color => #aaffcc

color and bgcolor can either be one of the 140 colors supported by all browsers, or a hexadecimal color in the standard #RRGGBB format.

Here is our colorized JSON grammar:

# Adapted from https://gist.github.com/pstuifzand/4447349
:start ::= json
json ::= object | array

object ::= '{' members '}' bgcolor => lightgoldenrodyellow
members ::= pair* proper => 1 separator => comma
pair ::= string ':' value

value ::= string
  | object
  | number color => purple
  | array
  | true color => blue
  | false color => blue
  | null color => tomato

array ::= '[' elements ']' bgcolor => lightcyan
elements ::= value* proper => 1 separator => comma

comma ~ ','

string ::= '"' in_string '"' color => peru
in_string ~ /([^"\\]|\\[\"\\\/bftnrt]|\\u[a-fA-F0-9]{4})*/
number ~ /-?([\d]+)(\.\d+)?([eE][+-]?\d+)?/

true ~ 'true'
false ~ 'false'
null ~ 'null'

:discard ~ whitespace
whitespace ~ [\s]+

This can be called from Crystal like so:

require "syntax"

grammar = <<-'END_EBNF'
# Grammar shown above
END_EBNF

input = %({"a" : 1, "b" : [1,2,3]})

highlighter = Syntax::Highlighter.new
highlighter.highlight(input, grammar)

which will produce an HTML document where the input is tagged with the specified colors.

This can be rendered directly, or other actions can be performed to provide useful information to the user.

# Why should I use this?

The given example above hopefully demonstrates how easy it is to describe a desired language and quickly achieve a good-looking result. In addition, the demo is fast enough to provide feedback in real-time.

The capability of this tool is thanks to Marpa, which makes it possible to describe potentially ambiguous grammars quickly and easily.

# What's the catch?

There are a significant number of languages that cannot be adequately described using only EBNF. As an example, consider Javascript. In Javascript, semicolons are optional, meaning that a liberal parser must be able to actively invent tokens in order to parse Javascript found in the wild.

The core library, marpa, does provide tools for handling these kinds of languages, however this functionality is not yet supported in syntax.cr.

# Conclusion

I think syntax.cr is likely the easiest way to go from a grammar to a nice-looking highlighter.

All the EBNF samples shown above were colorized using syntax.cr, and the demo should further show the capability of this tool.

For other applications, I would refer readers to the marpa library written in Crystal, or the original SLIF interface, written in Perl.

Omar Roth

The value of "AI" is mostly finite

# The Slop Trough

# So what are people actually building with this stuff?

# Given enough tokens, some bugs are shallow

# The consumer market will not backfill demand

# There won't be a one-person unicorn

# Looking forward

# Final thoughts

Wayland set the Linux Desktop back by 10 years

# Background and the problems with X11

# So what happened?

# Wayland is more secure (which means I can't do anything)

# Wayland is more performant (as long as you don't use NVidia)

# Wayland isn't just "one thing" (because Wayland doesn't do anything)

# Wayland is still under active development

# Opinion of Wayland developers

# Predictions and looking forward

# Things to be excited about

# Conclusion

# Additional reading

nanotrace: A nanosecond-scale profiler using Intel PT

# Introduction

# Overview of Intel PT

# Benchmarks

# magic-trace

# libxdc

# Optimizing nanotrace

# Caching block transitions

# Custom x86 disassembler

# Keep it simple, stupid

# Applications

# Profiling

# Fuzzing

# Debugging

# Final thoughts

# References

# Gallery

# ./hello (dynamically linked)

# ./hello (statically linked)

# lua -e print(1+2)

# cc1 hello.c

# clang hello.c

Instruction decoding at the speed of light

# The problem with x86

# Bounding compute and IO

# So, how does it perform in software?

# Back to basics

# Something (a bit) better

# Conclusion

# Some remaining questions

My website looks better than yours

# Aesthetics are an anti-signal

# Hey slopbot, make me a website

# LLMs are floor-raising, not bar-raising

# I wish it was as easy to make a website as a TUI

# In closing

Stepping away from open source

An introduction to syntax.cr

# Introduction

# Description of a language

# Colorizing a grammar

# Why should I use this?

# What's the catch?

# Conclusion

# `magic-trace`

# `libxdc`

# Optimizing `nanotrace`