Skip to content

Deterministic Thread Scheduler - pthread interposition for reproducible multi-threaded execution

License

Notifications You must be signed in to change notification settings

plasma-umass/dts

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

5 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

dts

Deterministic Thread Scheduler - a pthread interposition library for deterministic multithreaded execution. Enforces strict serialization so only one thread runs at a time, ensuring identical execution order across runs.

Features

  • Strict serialization: Only one thread executes at a time, eliminating race conditions
  • Deterministic scheduling: Threads are scheduled by deterministic thread ID (det_tid), ensuring repeatable execution order
  • Cross-platform: Supports macOS (DYLD_INSERT_LIBRARIES) and Linux (LD_PRELOAD)
  • Language agnostic: Works with any program using pthreads (C, C++, Java, Python, etc.)

Building

# Configure and build
cmake -B build
cmake --build build

# Build with debug logging
cmake -B build -DDTS_DEBUG_LOG=ON
cmake --build build

# Run tests
cd build && ctest

Usage

Basic Usage

# Run any program with deterministic threading
./scripts/run-dts.sh ./my_program

# Run with arguments
./scripts/run-dts.sh ./my_program arg1 arg2

# Run a Java application
./scripts/run-dts.sh java -jar myapp.jar

Java Applications

For maximum determinism with Java, use these flags:

./scripts/run-dts.sh java \
    -XX:+UseSerialGC \
    -XX:CICompilerCount=1 \
    -XX:-TieredCompilation \
    -jar myapp.jar
  • -XX:+UseSerialGC: Single-threaded garbage collector
  • -XX:CICompilerCount=1: Single JIT compiler thread
  • -XX:-TieredCompilation: Simpler compilation pipeline

Python Applications

For determinism with Python:

./scripts/run-dts.sh python3 my_script.py

# Disable hash randomization for additional determinism
PYTHONHASHSEED=0 ./scripts/run-dts.sh python3 my_script.py
  • PYTHONHASHSEED=0: Disables hash randomization (dict/set iteration order)
  • Note: Python's GIL already serializes most operations, but dts ensures deterministic ordering when threads contend

C/C++ Applications

For C/C++ programs, dts works out of the box:

./scripts/run-dts.sh ./my_program

For maximum determinism in your C/C++ code:

  • Avoid rand() or seed with a fixed value
  • Avoid reading system time for logic (only for logging)
  • Use deterministic memory allocators if allocation order matters
  • Avoid lock-free data structures (they bypass pthread primitives)

Linux CPU Pinning

On Linux, you can pin execution to a specific CPU for additional determinism:

# Pin to CPU 0 (default)
./scripts/run-dts-linux.sh ./my_program

# Pin to CPU 2
./scripts/run-dts-linux.sh --cpu 2 ./my_program

# No CPU pinning
./scripts/run-dts-linux.sh --no-pin ./my_program

Environment Variables

  • DTS_DEBUG=1: Enable debug logging (requires debug build)
  • DTS_CPU=N: Pin to CPU N (Linux only, with run-dts.sh)

How It Works

Interposition

  • macOS: Uses DYLD_INSERT_LIBRARIES with __DATA,__interpose section
  • Linux: Uses LD_PRELOAD with dlsym(RTLD_NEXT)

Deterministic Scheduling

  1. Each thread is assigned a deterministic TID (det_tid) at creation time
  2. Only one thread holds the "token" and can execute
  3. When a thread blocks (mutex, condvar, etc.), the token passes to the lowest det_tid in the run queue
  4. Wait queues are always sorted by det_tid, ensuring deterministic wake order

Shimmed Functions

Core Threading:

  • pthread_create, pthread_join, pthread_exit

Mutexes:

  • pthread_mutex_lock, pthread_mutex_trylock, pthread_mutex_unlock

Condition Variables:

  • pthread_cond_wait, pthread_cond_timedwait
  • pthread_cond_signal, pthread_cond_broadcast

Platform Notes

macOS

  • System Integrity Protection (SIP) blocks injection into Apple-signed binaries
  • For Java, use a non-system JVM: Homebrew OpenJDK, Azul Zulu, Amazon Corretto, etc.
  • CPU affinity is advisory only (THREAD_AFFINITY_POLICY is a hint)

Linux

  • No restrictions on LD_PRELOAD
  • True CPU pinning available via pthread_setaffinity_np() / taskset
  • Tested on Ubuntu, Fedora, and other major distributions

Architecture

┌─────────────────────────────────────────────────┐
│               Target Process                    │
├─────────────────────────────────────────────────┤
│  Shimmed pthread Interface                      │
│  (DYLD_INSERT_LIBRARIES / LD_PRELOAD)           │
├─────────────────────────────────────────────────┤
│  Deterministic Scheduler (token-based)          │
│  - Run queue (sorted by det_tid)                │
│  - Blocked pool (mutex/condvar waiters)         │
│  - Strict serialization: ONE thread at a time   │
├─────────────────────────────────────────────────┤
│  Platform Abstraction Layer                     │
│  ┌─────────────────┬─────────────────────────┐  │
│  │ macOS           │ Linux                   │  │
│  │ - Mach APIs     │ - pthread/condvar       │  │
│  │ - semaphore_t   │ - CPU affinity          │  │
│  └─────────────────┴─────────────────────────┘  │
├─────────────────────────────────────────────────┤
│  Original pthread + Kernel                      │
└─────────────────────────────────────────────────┘

Limitations

  • Performance: Strict serialization significantly reduces parallelism
  • Timeouts: Timed waits (pthread_cond_timedwait) are treated as regular waits for determinism
  • I/O: File and network I/O are not intercepted; external non-determinism can still occur
  • Signals: Signal handling is not shimmed

Verifying Determinism

Run the same program multiple times and compare execution traces:

for i in {1..5}; do
    DTS_DEBUG=1 ./scripts/run-dts.sh ./my_program 2>&1 | md5sum
done

All hashes should be identical if the program is deterministic.

License

MIT License. See LICENSE for details.

References

  • dthreads - Deterministic threading library (GPL-2.0)
  • Hermit - Deterministic Linux container (BSD-3)
  • rr - Record and replay debugger

About

Deterministic Thread Scheduler - pthread interposition for reproducible multi-threaded execution

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Contributors 2

  •  
  •