-
Notifications
You must be signed in to change notification settings - Fork 0
feat(postgres): bundle top-10 extensions in the appwrite/postgres image #2
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
abnegate
wants to merge
2
commits into
main
Choose a base branch
from
feat/dat-1666-postgres-extensions-image
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
2 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Some comments aren't visible on the classic Files Changed page.
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,40 @@ | ||
| name: Build and Verify | ||
|
|
||
| on: | ||
| pull_request: | ||
| push: | ||
| branches: | ||
| - main | ||
|
|
||
| jobs: | ||
| build: | ||
| name: Build and verify (PG ${{ matrix.pg_major }}) | ||
| runs-on: ubuntu-latest | ||
| strategy: | ||
| fail-fast: false | ||
| matrix: | ||
| pg_major: ['17', '18'] | ||
| steps: | ||
| - name: Checkout repository | ||
| uses: actions/checkout@v4 | ||
|
|
||
| - name: Set up Docker Buildx | ||
| uses: docker/setup-buildx-action@v3 | ||
|
|
||
| - name: Build image | ||
| uses: docker/build-push-action@v6 | ||
| with: | ||
| context: . | ||
| load: true | ||
| platforms: linux/amd64 | ||
| build-args: | | ||
| PG_MAJOR=${{ matrix.pg_major }} | ||
| tags: appwrite/postgres:ci-${{ matrix.pg_major }} | ||
| cache-from: type=gha,scope=pg-${{ matrix.pg_major }} | ||
| cache-to: type=gha,mode=max,scope=pg-${{ matrix.pg_major }} | ||
|
|
||
| - name: Verify extensions and preload libraries | ||
| env: | ||
| IMAGE: appwrite/postgres:ci-${{ matrix.pg_major }} | ||
| PG_MAJOR: ${{ matrix.pg_major }} | ||
| run: ./tests/verify.sh "$IMAGE" "$PG_MAJOR" | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,9 +1,18 @@ | ||
| FROM postgres:18 | ||
| ARG PG_MAJOR=18 | ||
|
|
||
| FROM postgres:${PG_MAJOR} | ||
|
|
||
| ARG PG_MAJOR | ||
|
|
||
| # hadolint ignore=DL3008 | ||
| RUN apt-get update && \ | ||
| apt-get upgrade -y && \ | ||
| apt-get install -y --no-install-recommends \ | ||
| postgresql-18-postgis-3 \ | ||
| postgresql-18-postgis-3-scripts \ | ||
| postgresql-18-pgvector && \ | ||
| rm -rf /var/lib/apt/lists/* | ||
| postgresql-${PG_MAJOR}-pgvector \ | ||
| postgresql-${PG_MAJOR}-postgis-3 \ | ||
| postgresql-${PG_MAJOR}-postgis-3-scripts \ | ||
| postgresql-${PG_MAJOR}-cron && \ | ||
| rm -rf /var/lib/apt/lists/* | ||
|
|
||
| RUN printf '\n%s\n' "shared_preload_libraries = 'pg_stat_statements,pg_cron'" \ | ||
| >> "/usr/share/postgresql/${PG_MAJOR}/postgresql.conf.sample" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,58 @@ | ||
| # appwrite/postgres | ||
|
|
||
| Custom PostgreSQL image for Appwrite Cloud dedicated databases (and the VectorsDB product). It is the official `postgres` image plus the ten most-used PostgreSQL extensions, bundled so they are available out of the box. | ||
|
|
||
| ## Bundled extensions | ||
|
|
||
| | Extension | `CREATE EXTENSION` name | Purpose | Preloaded | | ||
| | --------- | ----------------------- | ------- | --------- | | ||
| | pgvector | `vector` | Vector similarity search / embeddings | No | | ||
| | pg_stat_statements | `pg_stat_statements` | Per-query execution statistics | **Yes** | | ||
| | uuid-ossp | `uuid-ossp` | UUID generation | No | | ||
| | pgcrypto | `pgcrypto` | Hashing / encryption functions | No | | ||
| | pg_trgm | `pg_trgm` | Trigram fuzzy / similarity text search | No | | ||
| | PostGIS | `postgis` | Geospatial types, indexing, functions | No | | ||
| | citext | `citext` | Case-insensitive text | No | | ||
| | unaccent | `unaccent` | Accent-insensitive text search | No | | ||
| | hstore | `hstore` | Key/value pairs in a single column | No | | ||
| | pg_cron | `pg_cron` | In-database job scheduling | **Yes** | | ||
|
|
||
| Every extension is compiled into the image, so it appears in `pg_available_extensions` and installs with `CREATE EXTENSION IF NOT EXISTS`. | ||
|
|
||
| ## Preloaded libraries | ||
|
|
||
| `pg_stat_statements` and `pg_cron` require `shared_preload_libraries`, which must be set before the server starts. The image sets it in the cluster config template, so it applies to every initialized cluster with no runtime configuration: | ||
|
|
||
| ``` | ||
| shared_preload_libraries = 'pg_stat_statements,pg_cron' | ||
| ``` | ||
|
|
||
| `pg_cron` runs its scheduler in the default `postgres` database. | ||
|
|
||
| ## Supported major versions and tags | ||
|
|
||
| Built for each PostgreSQL major version Appwrite Cloud advertises (`Engine::getSupportedVersions()`): **17** and **18**. Publishing a release tag (e.g. `0.2.0`) produces, for each major: | ||
|
|
||
| | Tag | Meaning | | ||
| | --- | ------- | | ||
| | `appwrite/postgres:18`, `appwrite/postgres:17` | Floating tag for the major version | | ||
| | `appwrite/postgres:18-0.2.0`, `appwrite/postgres:17-0.2.0` | Immutable major + release | | ||
| | `appwrite/postgres:0.2.0`, `appwrite/postgres:latest` | Default major (18), for backward compatibility | | ||
|
|
||
| The default-major bare-semver tags keep the existing `version` -> image mapping working until the per-version wiring lands. | ||
|
|
||
| ## Build | ||
|
|
||
| ```bash | ||
| docker build --build-arg PG_MAJOR=18 -t appwrite/postgres:18 . | ||
| docker build --build-arg PG_MAJOR=17 -t appwrite/postgres:17 . | ||
| ``` | ||
|
|
||
| ## Verify | ||
|
|
||
| `tests/verify.sh` boots the image and asserts every extension is available, installs, and that both preload extensions actually load: | ||
|
|
||
| ```bash | ||
| docker build --build-arg PG_MAJOR=18 -t appwrite/postgres:ci-18 . | ||
| ./tests/verify.sh appwrite/postgres:ci-18 18 | ||
| ``` |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,122 @@ | ||
| #!/usr/bin/env bash | ||
| # | ||
| # Boots the built image and asserts every bundled extension is available, | ||
| # installable via CREATE EXTENSION, and that the two preload extensions | ||
| # (pg_stat_statements, pg_cron) actually load at startup. | ||
| # | ||
| # Usage: tests/verify.sh <image> <pg_major> | ||
| set -uo pipefail | ||
|
|
||
| IMAGE="${1:?image required}" | ||
| PG_MAJOR="${2:?pg major required}" | ||
| CONTAINER="verify-pg-${PG_MAJOR}-$$" | ||
| FAILED=0 | ||
|
|
||
| EXTENSIONS=( | ||
| vector | ||
| pg_stat_statements | ||
| uuid-ossp | ||
| pgcrypto | ||
| pg_trgm | ||
| postgis | ||
| citext | ||
| unaccent | ||
| hstore | ||
| pg_cron | ||
| ) | ||
|
|
||
| # shellcheck disable=SC2329 # invoked by trap | ||
| cleanup() { docker rm -f "$CONTAINER" >/dev/null 2>&1 || true; } | ||
| trap cleanup EXIT | ||
|
|
||
| fail() { echo "FAIL: $1"; FAILED=1; } | ||
|
|
||
| psql() { docker exec "$CONTAINER" psql -U postgres -d postgres -tAX -c "$1" 2>&1; } | ||
|
|
||
| docker run -d --name "$CONTAINER" \ | ||
| -e POSTGRES_PASSWORD=verify \ | ||
| -e POSTGRES_DB=postgres \ | ||
| "$IMAGE" >/dev/null | ||
|
|
||
| for _ in $(seq 1 60); do | ||
| docker exec "$CONTAINER" pg_isready -U postgres -d postgres >/dev/null 2>&1 && break | ||
| sleep 1 | ||
| done | ||
| if ! docker exec "$CONTAINER" pg_isready -U postgres -d postgres >/dev/null 2>&1; then | ||
| echo "FAIL: PostgreSQL never became ready" | ||
| docker logs "$CONTAINER" 2>&1 | tail -40 | ||
| exit 1 | ||
| fi | ||
| sleep 2 | ||
|
|
||
| echo "===== PG ${PG_MAJOR}: $(psql 'SHOW server_version;') =====" | ||
|
|
||
| echo "----- shared_preload_libraries -----" | ||
| spl="$(psql 'SHOW shared_preload_libraries;')" | ||
| echo "$spl" | ||
| if [[ "$spl" == *pg_stat_statements* && "$spl" == *pg_cron* ]]; then | ||
| echo "PASS: preload libraries set" | ||
| else | ||
| fail "shared_preload_libraries missing pg_stat_statements and/or pg_cron" | ||
| fi | ||
|
|
||
| echo "----- pg_available_extensions -----" | ||
| for ext in "${EXTENSIONS[@]}"; do | ||
| if [[ "$(psql "SELECT 1 FROM pg_available_extensions WHERE name = '$ext' LIMIT 1;")" == "1" ]]; then | ||
| echo "AVAILABLE: $ext" | ||
| else | ||
| fail "$ext not in pg_available_extensions" | ||
| fi | ||
| done | ||
|
|
||
| echo "----- CREATE EXTENSION IF NOT EXISTS -----" | ||
| for ext in "${EXTENSIONS[@]}"; do | ||
| result="$(psql "CREATE EXTENSION IF NOT EXISTS \"$ext\";")" | ||
| # psql -tA prints exactly "CREATE EXTENSION" on success. Anything else | ||
| # (an error, or empty output from a dead container) is a failure. | ||
| if [[ "$result" == "CREATE EXTENSION" ]]; then | ||
| echo "CREATED: $ext" | ||
| else | ||
| fail "CREATE EXTENSION $ext -> ${result:-<no output>}" | ||
| fi | ||
|
greptile-apps[bot] marked this conversation as resolved.
|
||
| done | ||
|
|
||
| echo "----- pg_stat_statements loaded -----" | ||
| if [[ "$(psql 'SELECT count(*) >= 0 FROM pg_stat_statements;')" == "t" ]]; then | ||
| echo "PASS: pg_stat_statements view queryable" | ||
| else | ||
| fail "pg_stat_statements view not queryable" | ||
| fi | ||
|
|
||
| echo "----- pg_cron loaded -----" | ||
| jobid="$(psql "SELECT cron.schedule('verify-job','* * * * *','SELECT 1');")" | ||
| if [[ "$jobid" =~ ^[0-9]+$ ]]; then | ||
| echo "PASS: pg_cron.schedule returned job $jobid" | ||
| psql "SELECT cron.unschedule($jobid);" >/dev/null | ||
| else | ||
| fail "pg_cron.schedule -> ${jobid:-<no output>}" | ||
| fi | ||
|
|
||
| echo "----- pgvector usable -----" | ||
| distance="$(psql "SELECT '[1,2,3]'::vector <-> '[3,2,1]'::vector;")" | ||
| if [[ -n "$distance" && "$distance" != *ERROR* ]]; then | ||
| echo "PASS: vector distance = $distance" | ||
| else | ||
| fail "vector operation -> $distance" | ||
| fi | ||
|
|
||
| echo "----- postgis usable -----" | ||
| postgis_version="$(psql 'SELECT postgis_version();')" | ||
| if [[ -n "$postgis_version" && "$postgis_version" != *ERROR* ]]; then | ||
| echo "PASS: postgis_version = $postgis_version" | ||
| else | ||
| fail "postgis_version() -> ${postgis_version:-<no output>}" | ||
| fi | ||
|
|
||
| echo | ||
| if [[ "$FAILED" == "0" ]]; then | ||
| echo "########## PG ${PG_MAJOR}: ALL CHECKS PASSED ##########" | ||
| else | ||
| echo "########## PG ${PG_MAJOR}: CHECKS FAILED ##########" | ||
| fi | ||
| exit "$FAILED" | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.