Version upgrade#8963
Open
DoDiODev wants to merge 23 commits into
Open
Conversation
…sions - upgrade backend dependencies (Go modules, gorm/mysql driver) and pin python/e2e deps - wrap MySQL dialector migrator to prevent duplicate PRIMARY KEY errors on AddColumn - refactor backend Docker setup (new streamlined Dockerfile, remove Dockerfile.local) and align build workflow dockerfile paths - upgrade config-ui stack (Node 24, Yarn 4, React 19, AntD 6, Vite 8, TypeScript 6) and regenerate lockfile - migrate to ESLint flat config and add Vitest setup/tests - patch miller-columns-select for React 19 compatibility and add regression coverage - switch app bootstrap to createRoot and apply related type-safety/runtime fixes - make Grafana path/proxy and compose Grafana DB settings configurable for local/devcontainer environments
…sions - upgrade backend dependencies (Go modules, gorm/mysql driver) and pin python/e2e deps - wrap MySQL dialector migrator to prevent duplicate PRIMARY KEY errors on AddColumn - refactor backend Docker setup (new streamlined Dockerfile, remove Dockerfile.local) and align build workflow dockerfile paths - upgrade config-ui stack (Node 24, Yarn 4, React 19, AntD 6, Vite 8, TypeScript 6) and regenerate lockfile - migrate to ESLint flat config and add Vitest setup/tests - patch miller-columns-select for React 19 compatibility and add regression coverage - switch app bootstrap to createRoot and apply related type-safety/runtime fixes - make Grafana path/proxy and compose Grafana DB settings configurable for local/devcontainer environments
…date Wave 1) Pin the previously floating `mysql:8` tag to the reproducible `mysql:8.4.10` (8.4 LTS line) across all developer/datasource compose files so local and devcontainer environments resolve to a deterministic MySQL version: - docker-compose-dev-mysql.yml - docker-compose.datasources.yml - .devcontainer/docker-compose.yml Align the swag CLI install in backend/Dockerfile.server from v1.16.1 to v1.16.6, matching backend/Makefile, backend/Dockerfile, backend/Dockerfile.local and the github.com/swaggo/swag v1.16.6 library in go.mod.
- grafana: bump base image 11.6.2 -> 12.4.5 (latest 12.x stable)
- grafana: drop grafana-piechart-panel install; dashboards use core "piechart"
- grafana: align dashboard pluginVersion metadata 11.6.2 -> 12.4.5 (144 refs, 13 files)
- config-ui: bump nginx-unprivileged 1.30.3 -> 1.31.2
- node.js stays on 24.x LTS (no change); node 26.x deferred
Smoke-tested: grafana image builds and starts, /api/health returns
{"database":"ok","version":"12.4.5"} with 0 error-level logs; dashboard JSON re-validated.
Move the Python runtime from EOL 3.9 (end-of-life 2025-10-31) to the active, security-supported 3.11 line. Runtime-only change: no Python library version bumps — Pydantic v1, SQLModel 0.0.8 and dbt 1.7 all run unchanged on 3.11 (library modernization in next step). Changes: - scripts-local/setup.sh: PYTHON_VERSION_REQUIRED 3.9 → 3.11, add PYTHON_PATCH_VERSION 3.11.9, rename ensure_python_39 → ensure_python_311 (pyenv 3.11.9) and update the call site. - backend/python/uv.sh: default DEVLAKE_PYTHON_VERSION 3.9 → 3.11. - pyproject.toml (pydevlake, azuredevops, fakeplugin): python "~3.9" → ">=3.11,<3.12"; re-locked all three poetry.lock files on 3.11. - backend/Dockerfile.server: python:3.9-slim-bookworm → python:3.11-slim-bookworm. - .devcontainer/devcontainer.json: python feature 3.9 → 3.11; node feature 18 → 24 (aligns with Node 24 LTS runtime, stale-pin fix). - Docs: update dependency_overview.csv (Python runtime + Node rows), restructure DEPENDENCY_UPDATE_PLAN.md into Wave 3a/3b and seed the Wave 4 "drop dbt-mysql → dbt-postgres 1.10" epic, and refresh backend/python/DevelopmentSetup.md. Validation: make build-pydevlake on 3.11.15; pytest 18 passed / 1 skipped (pydevlake + azuredevops + fakeplugin); shell + JSON syntax checks pass.
The Grafana 11→12 bump surfaced runtime regressions in the dashboard image.
Harden the entrypoint and remove broken duplicate dashboards.
entrypoint.sh:
- Resolve the home dashboard path for both layouts (baked-in image uses
/etc/grafana/dashboards/<mode>/Homepage.json, dev compose bind-mounts the
variant folder directly to /etc/grafana/dashboards). A wrong path made
Grafana 12 return HTTP 500 "Failed to load home dashboard".
- Self-heal read-only legacy volumes (uid 472, 0640) for the gid-0 runtime
user, else print an actionable remediation instead of a cryptic SQLite
"attempt to write a readonly database" crash.
- Remove the deprecated Angular grafana-piechart-panel from the data volume
(Grafana 12 dropped Angular support; dashboards use the core piechart).
- Default GF_SECURITY_ADMIN_PASSWORD to admin so the datasource bootstrap API
stops returning HTTP 401, and treat "already exists" as success (no 10x
retry loop on provisioned datasources).
dashboards:
- Remove grafana/dashboards/{mysql,postgresql}/ai-model-roi.json: mislabeled
duplicates of the PostgreSQL "AI Cost-Efficiency" dashboard. In the MySQL
deployment they referenced a non-existent "postgresql" datasource, which
fails the whole dashboard under Grafana 12's Scenes engine; the PostgreSQL
folder also had a duplicate uid collision.
Verified on dev mysql + postgresql compose: Grafana 12.4.5 healthy, home
dashboard 200, all 58 provisioned dashboards load, 0 level=error.
The convertIssues subtask failed with "Error 1406: Data too long for column component" because the domain issues.component column stayed varchar(255). The previous migration (20240813) targeted the wrong column name "components" (plural) and was never registered. - Add migration 20260629 to widen issues.component to text (MySQL + Postgres) - Register the migration in All() and fix the buggy 20240813 column name - Fix unhandled-error lint warnings in mysql_migrator.go - Document pending q_dev aws-sdk-go v1 -> v2 migration in TODO.md
Migrate the Python plugin framework from Pydantic v1 to v2 and bump supporting libraries. This is Wave 3b of the dependency modernization plan, split into two independent changes: — sqlglot bump (Grafana only): - grafana/scripts/requirements.txt: sqlglot 25.29.0 → 30.11.0 — Pydantic v2 / SQLModel migration: - pydantic ^1.10.2 → ^2.13, sqlmodel ^0.0.8 → ^0.0.38 - inflect ^6.0.2 → ^7.5, fire ^0.4.0 → ^0.7 - pytest ^7.2.2 → ^9.1, jsonpointer ^2.3 → ^3.1 Key migration changes: - model.py: @validator → @field_validator(mode='before'), class Config → model_config = ConfigDict(...), sa_column=Column(...) → sa_type=DateTime()/Text(), explicit default=None on all Optional fields - message.py: const=True → Literal, __root__ → RootModel - model_info.py: .schema() → .model_json_schema(), definitions → $defs, flatten Pydantic v2 anyOf for Optional fields to plain "type" string (Go backend's conversion.go requires type as a string, not an array) - extractor.py: __fields__ → model_fields, field.field_info.extra → field_info.json_schema_extra, added _is_optional() annotation check - __init__.py: Field wrapper sets json_schema_extra manually (SQLModel 0.0.38 drops schema_extra from FieldInfo) - Serialization: .json() → .model_dump_json(), .dict() → .model_dump() All 10 pydevlake tests pass. azuredevops and fakeplugin imports verified. dbt-postgres/dbt-mysql pins intentionally untouched
Bump PostgreSQL images to 18.1: - docker-compose-dev-postgresql.yml: postgres:17.2 → postgres:18.1 - backend/test/e2e/remote/docker-compose.test.yml: postgres:14.2-alpine → postgres:18.1-alpine
Image & entrypoint:
- Bump base image to grafana/grafana:13.0.2
- Change PG datasource type from "postgres" to "grafana-postgresql-datasource"
(plugin renamed in Grafana 13)
- Add "database" to jsonData (required by new PG plugin)
- Delete ALL existing datasources by ID on startup (handles read-only
legacy-provisioned datasources from old volumes)
- postgresVersion 1400 → 1800
- Comments "Grafana 12" → "Grafana 12+"
Dashboard JSON files:
- Align pluginVersion to "13.0.2" (1267 occurrences)
- Replace deprecated name-string datasource references with UID-based objects:
- postgresql: {"type": "grafana-postgresql-datasource", "uid": "devlake-postgres-api"}
- mysql: {"type": "mysql", "uid": "devlake-mysql-api"}
Validated on both MySQL (:3001) and PostgreSQL (:3002) compose variants:
- /api/health → version 13.0.2
- Datasource health → "Database Connection OK"
- 58 dashboards load, 0 level=error logs
Author
|
Labels: dependencies, infrastructure, no-breaking-change |
Updated Dockerfile.local to use build caches for improved build performance.
Refactor Dockerfile to support multi-platform builds and optimize dependencies.
The convertIssueCodeBlocks subtask failed with: Error 1406 (22001): Data too long for column 'component' at row 91 Root cause: The domain layer table cq_issue_code_blocks had component as varchar(256) (GORM default), while SonarQube component paths can exceed this limit. The tool layer had varchar(500) which was also insufficient. Changes: - Change CqIssueCodeBlock.Component to gorm:"type:text" (domain layer) - Change SonarqubeIssueCodeBlock.Component to gorm:"type:text" (tool layer) - Add migration to alter cq_issue_code_blocks.component to TEXT - Add migration to drop index and alter _tool_sonarqube_issue_code_blocks.component to TEXT - Add unit tests verifying long component handling and GORM tag correctness - Add E2E test reproducing the original error with >256 and >500 char paths
- Migration scripts: safely drop indexes before changing component columns to TEXT, ignoring errors if index does not exist (supports both MySQL and PostgreSQL) - CI build workflow: use Dockerfile.server instead of Dockerfile
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
pr-type/bug-fix,pr-type/feature-development, etc.Summary
Dependency Modernization & Infrastructure Hardening
Comprehensive update of all runtime dependencies, container images, and developer
tooling to current LTS/stable versions. Goal: bring every component to a supported
version line, harden the Grafana deployment for production use, and reduce
technical debt.
Summary of Changes
🖥️ Frontend (config-ui)
Dockerfile, Volta, and corepack
eslint.config.mjs); replacedunmaintained
eslint-plugin-headerwitheslint-plugin-headerstscerrors introduced by the major bumps (React 19 refs/useRef,react-markdown v10 API, cron-parser v5 API, missing
@types/lodash)node:24-bookworm-slim, nginx1.31.2🐍 Python (backend/python)
pyproject.toml,Dockerfile.server,.devcontainer)validators, model configs, serialization, schema generation)
exclude_none) verified byte-identical— no breaking change for the Go↔Python contract
📊 Grafana
grafana/scripts/entrypoint.sh):grafana-piechart-panelauto-removaldelete-retry for stale provisioned datasources
ai-model-roi.json— uid collision +cross-datasource reference failures in Grafana 12+ Scenes engine)
pluginVersionaligned to"13.0.2"across all 58 dashboardspostgresVersion1400 → 1800, type changed tografana-postgresql-datasource(Grafana 13 rename), UID-based references🗄️ Databases
docker-compose-dev-postgresql.yml, e2e test compose)mysql:8tomysql:8.4.10🐳 Container Images & Infra
config-ui/Dockerfile)grafana-cli plugins install grafana-piechart-panel(core pie chart used; Angular plugin dropped in Grafana 12)
Not Included (planned follow-ups)
Commit Structure
Changes are grouped by phase, each independently committed and buildable:
nginx-unprivileged1.30 → 1.31Validation
yarn install && yarn tsc && yarn lint && yarn build && yarn testpoetry run pytest(pydevlake: 10 tests)/api/health, 58 dashboards load, 0level=errorFORCE_MIGRATION=true,/ready=200make build, unit tests, E2E (AE plugin)Does this close any open issues?
no
Screenshots
Include any relevant screenshots here.
Other Information
Any other information that is important to this PR.