reorganize code into submodules (#106)

* move api code to submodule

* add symlink

* fix symlink

* fastapi_stac -> stac_fastapi

* update symlink

* add app routes, update imports

* rename

* add backend submodule

* add symlink

* update imports

* add extensions submodule, move core extensions from api

* add extensions symlink

* move bulk transactions to third party extensions

* move tiles extension to third_party

* backend.client -> backend.core

* define third party clients next to the extension

* remove third party clients from core

* correct package names

* create postgres submodule, move the sqlalchemy code

* move config

* update extension imports

* move errors

* move openapi

* fix tests

* add types submodule for shared types, finish moving db code, save progress on seperation of concerns...

* lint everything, remove mypy for now

* update pytest.ini

* Updates to submodules PR (#109)

* Consolidate image building to only use docker-compose

The makefile was building a named image, which was the same image as
the docker-compose app service image, but they were not the same
name. This commit consolidates the building and running of the image
through the app service of the docker-compose, and absorbes the
.env.example file into the env setup in docker-compose.yml

* Use docker network instead of external hosts.

* Add stac_fastapi_server package.

This brings in the logic of stac_api.app into it's own
subpackage. This is used by the development environment to spin up
stac-fastapi via docker-compose

* Read joplin data locally.

The remotely hosted joplin data was failing the ingest as they do not
have a "links" property, which makes them invalid STAC - the previous
server didn't complain, but that was breaking the ingest into
stac-fastapi.

This adds an empty link properties to the data downloaded from the remote location.

* Add fields to ApiSettings.

This pulls in the settings fields taken from
stac_api.config.ApiSettings.

* Don't build separate image for migration.

The migration container uses the same image as the app, but before
this change a new image was being built off the same docker
container. This moves migraitons to both depend on app and use it's
image.
It also mounts the source into the volume in case the dev is testing
changes to migrations or the ingest script.

* Pip install subpackages in the Dockerfile

* Structure dependencies in setup.py's

Restructure the Dockerfile to copy only what it needs as it installs
the subpackages. This will help catch dependency errors.

* Delete old stac_api package

* Add version to stac-fastapi-server

* Add a publish script.

This will be used by CI to publish on tags.

* Add publish GitHub action.

This will publish stac-fastapi to pypi on any tag.

* Fix broken import

* Pull top level package version from api version

* Pin stac-pydantic to 1.3.8

* Move search to polygon method out of types.

Has dependency on shapely, which can be avoided in the types subproject.

* Use Optional in param as it's possible to pass None

* Refactor set logic

* Pin sqlalchemy to 1.3.23

* Move global settings registry into stac-fastapi.types

Settings are used in the types subproject, which is the base
layer. This change makes the global instance of the settings that is
set by the API to be held in the types subproject to avoid circular dependencies.

* Move logic to include query fields in Fields to postgres client.

This logic requires settings that are specific to the postgres
client. Move this logic into the postgres client subpackage and remove
from the pydantic validator chain.

* Fixes from pre-commit formatting and linting

* lint

* rename postgres package to sqlalchemy

* rename PostgresSettings to SqlalchemySettings

* update readme

* relock types pipfile

Co-authored-by: Rob Emanuele <rdemanuele@gmail.com>

Author: geospatial-jeff

.env.example

@@ -0,0 +1,30 @@
+name: Publish
+
+on:
+  push:
+    tags:
+      - "*"
+
+jobs:
+  release:
+    name: release
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v2
+
+      - name: Set up Python 3.x
+        uses: actions/setup-python@v2
+        with:
+          python-version: "3.x"
+
+      - name: Install release dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install setuptools wheel twine
+
+      - name: Build and publish package
+        env:
+          TWINE_USERNAME: ${{ secrets.PYPI_STACUTILS_USERNAME }}
+          TWINE_PASSWORD: ${{ secrets.PYPI_STACUTILS_PASSWORD }}
+        run: |
+          scripts/publish

.github/workflows/publish.yml

@@ -34,13 +34,13 @@ repos:
                  '--select=D1',
                  # Don't require docstrings for tests
                  '--match=(?!test).*\.py']
-    -
-        repo: https://github.com/pre-commit/mirrors-mypy
-        rev: v0.770
-        hooks:
-            -   id: mypy
-                language_version: python3.8
-                args: [--no-strict-optional, --ignore-missing-imports]
+#    -
+#        repo: https://github.com/pre-commit/mirrors-mypy
+#        rev: v0.770
+#        hooks:
+#            -   id: mypy
+#                language_version: python3.8
+#                args: [--no-strict-optional, --ignore-missing-imports]
     -
         repo: https://github.com/PyCQA/pydocstyle
         rev: 5.1.1

.pre-commit-config.yaml

@@ -11,17 +11,45 @@ ENV CURL_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt
 
 ARG install_dev_dependencies=true
 
-WORKDIR /app
+RUN mkdir -p /app
+
+# Install stac_fastapi.types
+COPY ./stac_fastapi_types /app/stac_fastapi_types
+WORKDIR /app/stac_fastapi_types
+RUN pipenv install --deploy --ignore-pipfile ${install_dev_dependencies:+--dev}
+
+# Install stac_api.extensions.
+COPY ./stac_fastapi_extensions /app/stac_fastapi_extensions
+WORKDIR /app/stac_fastapi_extensions
+RUN pipenv install --deploy --ignore-pipfile ${install_dev_dependencies:+--dev}
+
+# Install stac_api.api.
+COPY ./stac_fastapi_api /app/stac_fastapi_api
+WORKDIR /app/stac_fastapi_api
+RUN pipenv install --deploy --ignore-pipfile ${install_dev_dependencies:+--dev}
+
+# Install stac_api.sqlalchemy.
+COPY ./stac_fastapi_sqlalchemy /app/stac_fastapi_sqlalchemy
+WORKDIR /app/stac_fastapi_sqlalchemy
+RUN pipenv install --deploy --ignore-pipfile ${install_dev_dependencies:+--dev}
 
-COPY Pipfile Pipfile.lock setup.py ./
+# Install stac_api.server.
+COPY ./stac_fastapi_server /app/stac_fastapi_server
+WORKDIR /app/stac_fastapi_server
 RUN pipenv install --deploy --ignore-pipfile ${install_dev_dependencies:+--dev}
 
-COPY . ./
+# Install base package
+COPY ./setup.py /app/
+COPY ./Pipfile /app/
+COPY ./Pipfile.lock /app/
+WORKDIR /app
+RUN pipenv install --deploy --ignore-pipfile ${install_dev_dependencies:+--dev}
 
 ENV APP_HOST=0.0.0.0
 ENV APP_PORT=80
 ENV RELOAD="true"
 
 ENTRYPOINT ["pipenv", "run"]
-CMD if [ "$RELOAD" ]; then uvicorn stac_api.app:app --host=${APP_HOST} --port=${APP_PORT} --reload ; \
-    else gunicorn stac_api.app:app --preload -k uvicorn.workers.UvicornWorker --bind ${APP_HOST}:${APP_PORT} --log-level info; fi
+
+CMD if [ "$RELOAD" ]; then uvicorn stac_fastapi.server.app:app --host=${APP_HOST} --port=${APP_PORT} --reload ; \
+    else gunicorn stac_fastapi.server.app:app --preload -k uvicorn.workers.UvicornWorker --bind ${APP_HOST}:${APP_PORT} --log-level info; fi

Dockerfile

@@ -2,17 +2,15 @@
 APP_HOST ?= 0.0.0.0
 APP_PORT ?= 8080
 EXTERNAL_APP_PORT ?= ${APP_PORT}
-run_docker = docker run -it --rm \
+run_docker = docker-compose run --rm \
 				-p ${EXTERNAL_APP_PORT}:${APP_PORT} \
-				-v $(shell pwd):/app \
-				--env APP_HOST=${APP_HOST} \
-				--env APP_PORT=${APP_PORT} \
-				--env-file .env.example \
-				stac-utils/stac-api:latest
+				-e APP_HOST=${APP_HOST} \
+				-e APP_PORT=${APP_PORT} \
+				app
 
 .PHONY: image
 image:
-	docker build -t stac-utils/stac-api:latest .
+	docker-compose build app
 
 .PHONY: docker-run
 docker-run: image

Makefile

@@ -1,20 +1,17 @@
 
 <p align="center">
-  <p align="center">Arturo's STAC compliant API implementation.</p>
+  <p align="center">sta.</p>
 </p>
 
 <p align="center">
-  <a href="https://github.com/stac-utils/arturo-stac-api/actions?query=workflow%3Acicd" target="_blank">
-      <img src="https://github.com/stac-utils/arturo-stac-api/workflows/arturo-stac-api/badge.svg" alt="Test">
+  <a href="https://github.com/stac-utils/stac-fastapi/actions?query=workflow%3Acicd" target="_blank">
+      <img src="https://github.com/stac-utils/stac-fastapi/workflows/stac-fastapi/badge.svg" alt="Test">
   </a>
-  <a href="https://codecov.io/gh/stac-utils/arturo-stac-api" target="_blank">
-      <img src="https://codecov.io/gh/stac-utils/arturo-stac-api/branch/master/graph/badge.svg" alt="Coverage">
+  <a href="https://pypi.org/project/stac-fastapi" target="_blank">
+      <img src="https://img.shields.io/pypi/v/stac-fastapi?color=%2334D058&label=pypi%20package" alt="Package version">
   </a>
-  <a href="https://pypi.org/project/arturo-stac-api" target="_blank">
-      <img src="https://img.shields.io/pypi/v/arturo-stac-api?color=%2334D058&label=pypi%20package" alt="Package version">
-  </a>
-  <a href="https://github.com/stac-utils/arturo-stac-api/blob/master/LICENSE" target="_blank">
-      <img src="https://img.shields.io/github/license/stac-utils/arturo-stac-api.svg" alt="Downloads">
+  <a href="https://github.com/stac-utils/stac-fastapi/blob/master/LICENSE" target="_blank">
+      <img src="https://img.shields.io/github/license/stac-utils/stac-fastapi.svg" alt="Downloads">
   </a>
 </p>
 
@@ -23,59 +20,24 @@
 
 ---
 
-**Documentation**: [https://stac-utils.github.io/arturo-stac-api/](https://stac-utils.github.io/arturo-stac-api/)
+**Documentation**: [https://stac-utils.github.io/stac-fastapi/](https://stac-utils.github.io/stac-fastapi/)
 
-**Source Code**: [https://github.com/stac-utils/arturo-stac-api](https://github.com/stac-utils/arturo-stac-api)
+**Source Code**: [https://github.com/stac-utils/stac-fastapi](https://github.com/stac-utils/stac-fastapi)
 
 ---
 
-Python library for building a STAC compliant FastAPI application.  It provides:
-- An API layer which enforces the [stac-api spec](https://github.com/radiantearth/stac-api-spec) and allows users
-to customize how the API interacts with their data through dependency injection.
-- A PostGIS implementation using [sqlalchemy](https://github.com/sqlalchemy/sqlalchemy)/[geoalchemy2](https://geoalchemy-2.readthedocs.io/en/latest/).
+Python library for building a STAC compliant FastAPI application.  The project is split up into several namespace
+packages:
+- **stac_fastapi.api**: An API layer which enforces the [stac-api-spec](https://github.com/radiantearth/stac-api-spec).
+- **stac_fastapi.extensions**: Abstract base classes for [STAC API extensions](https://github.com/radiantearth/stac-api-spec/blob/master/extensions.md) and third-party extensions.
+- **stac_fastapi.server**: Standalone FastAPI server for the application.
+- **stac_fastapi.sqlalchemy**: Postgres backend implementation with sqlalchemy.
+- **stac_fastapi.types**: Shared types and abstract base classes used by the library.
 
 Initially developed by [arturo-ai](https://github.com/arturo-ai).
 
 ```
-pip install arturo-stac-api
-```
-
-## Usage
-```python
-from stac_api.api.app import StacApi
-from stac_api.clients.postgres.core import CoreCrudClient
-from stac_api.clients.postgres.session import Session
-from stac_api.config import ApiSettings
-
-settings = ApiSettings()
-api = StacApi(
-    settings=settings,
-    client=CoreCrudClient(
-        session=Session(settings.reader_connection_string, settings.writer_connection_string)
-    ),
-)
-
-# FastAPI application
-app = api.app
-```
-
-## Project Structure
-```
-.
-├── alembic             # Database migrations
-│   └── versions        # Migration versions
-├── scripts             # Scripts for local development
-├── stac_api
-│   ├── api             # API layer
-│   ├── clients
-│   │   ├── postgres    # Postgres CRUD client
-│   │   └── tiles       # OGC Tiles API client
-│   ├── models          # Pydantic and ORM models
-│   └── utils           # Helper functions
-└── tests
-    ├── api             # Test api creation
-    ├── clients         # Test application logic
-    └── resources       # Test api endpoints
+pip install stac-fastapi
 ```
 
 ## Local Development

README.md

@@ -28,7 +28,8 @@
 
 def get_connection_url() -> str:
     """
-    Get connection URL from environment variables (see `.env.example`)
+    Get connection URL from environment variables
+    (see environment variables set in docker-compose)
     """
     postgres_user = os.environ["POSTGRES_USER"]
     postgres_pass = os.environ["POSTGRES_PASS"]

alembic/env.py

@@ -3,6 +3,7 @@ version: '3'
 services:
   app:
     container_name: stac-api
+    image: stac-utils/stac-api
     build:
       context: .
       dockerfile: Dockerfile
@@ -14,17 +15,19 @@ services:
       - POSTGRES_USER=username
       - POSTGRES_PASS=password
       - POSTGRES_DBNAME=postgis
-      - POSTGRES_HOST_READER=host.docker.internal
-      - POSTGRES_HOST_WRITER=host.docker.internal
-      - POSTGRES_PORT=5439
+      - POSTGRES_HOST_READER=database
+      - POSTGRES_HOST_WRITER=database
+      - POSTGRES_PORT=5432
+      - WEB_CONCURRENCY=10
+      - VSI_CACHE=TRUE
+      - GDAL_HTTP_MERGE_CONSECUTIVE_RANGES=YES
+      - GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR
     ports:
       - "8081:8081"
     volumes:
       - ./:/app
     depends_on:
       - database
-    extra_hosts:
-      - "host.docker.internal:host-gateway"
 
   database:
     container_name: stac-db
@@ -37,19 +40,22 @@ services:
       - "5439:5432"
 
   migration:
-    build:
-      context: .
-      dockerfile: Dockerfile
+    image: stac-utils/stac-api
     environment:
       - ENVIRONMENT=development
       - POSTGRES_USER=username
       - POSTGRES_PASS=password
       - POSTGRES_DBNAME=postgis
-      - POSTGRES_HOST=host.docker.internal
-      - POSTGRES_PORT=5439
+      - POSTGRES_HOST=database
+      - POSTGRES_PORT=5432
+    volumes:
+      - ./:/app
     command: >
       bash -c "sleep 10 && alembic upgrade head && python scripts/ingest_joplin.py"
     depends_on:
       - database
-    extra_hosts:
-      - "host.docker.internal:host-gateway"
+      - app
+
+networks:
+  default:
+    name: stac-fastapi-network

docker-compose.yml

@@ -1,3 +1,3 @@
 [pytest]
 testpaths = tests
-addopts = -sv --cov=stac_api --cov-fail-under=85 --cov-report=term-missing
+addopts = -sv

pytest.ini

@@ -1,24 +1,27 @@
 """Ingest sample data during docker-compose"""
-
+import json
+import os
 from urllib.parse import urljoin
 
 import requests
 
 bucket = "arturo-stac-api-test-data"
-app_host = "http://host.docker.internal:8081"
+app_host = "http://app:8081"
+
 
+def ingest_joplin_data(data_dir="/app/tests/data/joplin"):
+    """ingest data."""
 
-def ingest_joplin_data():
-    """ingest data"""
-    r = requests.get(f"https://{bucket}.s3.amazonaws.com/joplin/collection.json")
-    collection = r.json()
+    with open(os.path.join(data_dir, "collection.json")) as f:
+        collection = json.load(f)
 
     r = requests.post(urljoin(app_host, "/collections"), json=collection)
     if r.status_code not in (200, 409):
         r.raise_for_status()
 
-    r = requests.get(f"https://{bucket}.s3.amazonaws.com/joplin/index.geojson")
-    index = r.json()
+    with open(os.path.join(data_dir, "index.geojson")) as f:
+        index = json.load(f)
+
     for feat in index["features"]:
         del feat["stac_extensions"]
         r = requests.post(