Skip to content

Git Sync

Synchronize DAG definitions with a Git repository.

Configuration

yaml
gitSync:
  enabled: true
  repository: github.com/your-org/dags
  branch: main
  path: ""  # Subdirectory in repo (empty = root)
  pushEnabled: true

  auth:
    type: token
    token: ${GITHUB_TOKEN}

  autoSync:
    enabled: true
    onStartup: true
    interval: 300  # seconds

  commit:
    authorName: Dagu
    authorEmail: dagu@localhost

Authentication

Token (HTTPS)

GitHub offers two types of Personal Access Tokens. Choose one:

Fine-grained tokens are scoped to specific repositories with granular permissions.

Create at: GitHub > Settings > Developer settings > Personal access tokens > Fine-grained tokens

Required settings:

SettingValue
Repository accessSelect the repository containing your DAGs
Permissions > Repository permissions > ContentsRead and write (for pull and push)
Permissions > Repository permissions > MetadataRead (required for all fine-grained tokens)

If you only need to pull (read-only mode with pushEnabled: false):

SettingValue
Permissions > Repository permissions > ContentsRead-only
Permissions > Repository permissions > MetadataRead

Option B: Classic PAT

Classic tokens use broad OAuth scopes and access all repositories you can access.

Create at: GitHub > Settings > Developer settings > Personal access tokens > Tokens (classic)

Required scopes:

Repository TypeRequired ScopeWhat it grants
Private repositoryrepoFull access to all private and public repos you can access
Public repository onlypublic_repoRead/write access to public repos only

WARNING

Classic PATs with repo scope grant access to ALL your private repositories, not just the DAGs repository. Use fine-grained tokens for better security.

Configuration

yaml
gitSync:
  repository: github.com/your-org/dags
  branch: main
  auth:
    type: token
    token: ${GITHUB_TOKEN}

Environment variables:

bash
export DAGU_GITSYNC_AUTH_TYPE=token
export DAGU_GITSYNC_AUTH_TOKEN=ghp_xxxxxxxxxxxx

SSH Key

Supported key types: RSA, ED25519, ECDSA (via go-git/v5).

Repository URL must use SSH format: git@github.com:org/repo.git

Add your public key to GitHub: Settings > SSH and GPG keys > New SSH key

yaml
gitSync:
  repository: git@github.com:your-org/dags.git
  branch: main
  auth:
    type: ssh
    sshKeyPath: /home/user/.ssh/id_ed25519
    sshPassphrase: ${SSH_PASSPHRASE}  # Only if key is encrypted

Environment variables:

bash
export DAGU_GITSYNC_AUTH_TYPE=ssh
export DAGU_GITSYNC_AUTH_SSH_KEY_PATH=/home/user/.ssh/id_ed25519
export DAGU_GITSYNC_AUTH_SSH_PASSPHRASE=your-passphrase

CLI Commands

Status

bash
dagu sync status

Output: repository, branch, last sync time/commit, DAG counts by status.

Pull

bash
dagu sync pull

Fetches remote changes. Local modifications preserved (marked modified). Conflicts detected when both local and remote changed.

Publish

bash
dagu sync publish my-dag -m "Updated schedule"
dagu sync publish --all -m "Batch update"

Flags:

  • -m, --message <string>: Commit message
  • --all: Publish all modified DAGs
  • -f, --force: Force publish (overwrite conflicts)

Discard

bash
dagu sync discard my-dag

Restores remote version. Permanently discards local changes.

DAG Status States

StatusMeaning
syncedLocal matches remote
modifiedLocal has unpublished changes
untrackedExists locally only
conflictBoth local and remote modified since last sync

Conflict Resolution

bash
# Option 1: Force publish (overwrites remote)
dagu sync publish my-dag --force -m "Override remote"

# Option 2: Discard local (restores remote)
dagu sync discard my-dag

# Option 3: Manual merge, then publish
# Edit the file, then:
dagu sync publish my-dag -m "Merged changes"

REST API

MethodEndpointDescription
GET/api/v2/sync/statusOverall sync status
POST/api/v2/sync/pullPull from remote
POST/api/v2/sync/publish-allPublish all modified DAGs
POST/api/v2/sync/test-connectionTest remote connection
GET/api/v2/sync/configGet configuration
PUT/api/v2/sync/configUpdate configuration
POST/api/v2/dags/{name}/sync/publishPublish single DAG
POST/api/v2/dags/{name}/sync/discardDiscard single DAG changes

Examples

bash
# Get status
curl http://localhost:8080/api/v2/sync/status

# Pull
curl -X POST http://localhost:8080/api/v2/sync/pull

# Publish DAG
curl -X POST http://localhost:8080/api/v2/dags/my-dag/sync/publish \
  -H "Content-Type: application/json" \
  -d '{"message": "Updated schedule", "force": false}'

Configuration Reference

Main Options

OptionTypeDefaultEnvironment Variable
enabledboolfalseDAGU_GITSYNC_ENABLED
repositorystring(required)DAGU_GITSYNC_REPOSITORY
branchstringmainDAGU_GITSYNC_BRANCH
pathstring""DAGU_GITSYNC_PATH
pushEnabledbooltrueDAGU_GITSYNC_PUSH_ENABLED

Authentication (auth)

OptionTypeDefaultEnvironment Variable
typestringtokenDAGU_GITSYNC_AUTH_TYPE
tokenstring-DAGU_GITSYNC_AUTH_TOKEN
sshKeyPathstring-DAGU_GITSYNC_AUTH_SSH_KEY_PATH
sshPassphrasestring-DAGU_GITSYNC_AUTH_SSH_PASSPHRASE

Auto-Sync (autoSync)

OptionTypeDefaultEnvironment Variable
enabledboolfalseDAGU_GITSYNC_AUTOSYNC_ENABLED
onStartupbooltrueDAGU_GITSYNC_AUTOSYNC_ON_STARTUP
intervalint300DAGU_GITSYNC_AUTOSYNC_INTERVAL

Commit (commit)

OptionTypeDefaultEnvironment Variable
authorNamestringDaguDAGU_GITSYNC_COMMIT_AUTHOR_NAME
authorEmailstringdagu@localhostDAGU_GITSYNC_COMMIT_AUTHOR_EMAIL

Read-Only Mode

Set pushEnabled: false for deployments where DAGs are managed externally (CI/CD pushes to repo, Dagu only pulls).

yaml
gitSync:
  enabled: true
  repository: github.com/your-org/dags
  branch: main
  pushEnabled: false
  auth:
    type: token
    token: ${GITHUB_TOKEN}
  autoSync:
    enabled: true
    interval: 300

In this mode, dagu sync publish returns error "push operations are disabled".

Permissions

Git Sync operations require the following user roles (when authentication is enabled):

OperationRequired Role
View statusAny authenticated user
View configAny authenticated user
Test connectionAny authenticated user
Pull changesadmin or manager
Publish DAG(s)admin or manager
Discard changesadmin or manager
Update configadmin only

Additionally:

  • Server must have writeDAGs permission enabled in server.permissions
  • If pushEnabled: false, all write operations return 403 Forbidden

Data Management

Storage Locations

LocationPathPurpose
Local DAGs{dagsDir}/User-editable DAG files
State file{dataDir}/gitsync/state.jsonTracks sync status per DAG
Repo cache{dataDir}/gitsync/repo/Shallow clone of remote (depth=1)

State File Format

Top-level fields:

FieldTypeDescription
versionintFormat version (currently 1)
repositorystringRemote URL
branchstringTracked branch
lastSyncAttimestampLast successful sync time
lastSyncCommitstringGit commit hash from last sync
lastSyncStatusstring"success" or "error"
lastErrorstringError message (if failed)
dagsmapDAG ID → DAGState

Per-DAG state (dags[id]):

FieldTypeDescription
statusstringsynced, modified, untracked, conflict
baseCommitstringCommit hash when last synced
lastSyncedHashstringContent hash at last sync (sha256:hex)
localHashstringCurrent local content hash
lastSyncedAttimestampWhen DAG was last synced
modifiedAttimestampWhen local file was modified
remoteCommitstringConflicting commit (if conflict)
remoteAuthorstringConflict commit author
remoteMessagestringConflict commit message
conflictDetectedAttimestampWhen conflict was detected

Change Detection

Content hash: sha256: + hex(SHA256(file_content))

  • localHash == lastSyncedHashsynced
  • localHash != lastSyncedHashmodified
  • No lastSyncedHashuntracked

Sync Algorithm

Pull:

  1. git fetch --depth=1 from remote
  2. For each remote .yaml/.yml file:
    • Compute remote hash
    • If local status=modified AND remote hash != lastSyncedHash → conflict
    • Else update local file, set status=synced
  3. Scan local DAGs dir for untracked files
  4. Save state.json

Publish:

  1. If status=conflict and force=false → error
  2. Copy DAG file to repo working tree
  3. git add + git commit (with configured author)
  4. git push
  5. Update state: status=synced, update hashes

Conflict occurs when:

  • Local file modified (localHash != lastSyncedHash)
  • AND remote changed (pulled hash != lastSyncedHash)

State Transitions

[new file] → untracked
                ↓ publish
            synced ←──────────────┐
                ↓ local edit      │
            modified              │
           ↙        ↘             │
    [pull+remote    [publish] ────┘
     changed]

    conflict
        ↓ publish --force OR discard
    synced

Internals

ItemDetail
Git librarygo-git/v5
Synced files.yaml, .yml only
Clone depthShallow clone (depth: 1)

References