Problem Description

Querybook currently models database schemas and tables as separate objects and constructs fully qualified table names by joining schema and table (e.g.,<schema>.<table>). However, many query engines and metastores support catalogs and three-part identifiers (<catalog>.<schema>.<table>).

Currently, running queries with three-part table names works if the Query Engine supports it, but there’s no support in the metastore, table browser, lineage, etc. This limitation affects how table names are parsed, stored, displayed, and searched across both the front-end and back-end.

Expected End Goal

Introduce Catalog Support: Implement a new Catalog concept in Querybook, updating both back-end and front-end to support three-level identifiers.

• Backward Compatibility: Ensure that the system continues to work with engines that do not use catalogs by making the catalog component optional, both in the metastore sync code and the UI.

• Seamless Integration: Update all relevant code paths (parsing, construction, filtering, validation, search, and display) to handle both two-part and three-part table names.

• Database Changes: Add a new DataCatalog table linked to DataSchema, allowing for catalog-specific metadata and relationships.

• Metastore Loader Changes: Support optionally loading catalogs, with fallback support for loaders that don’t support catalogs.

• Consistent User Experience: Maintain or improve current use cases (table lookup, autocomplete, ACL validation, lineage tracking, search, AI context) with the new catalog-aware model.

Sample Screenshots

Catalog support example on the table list.

Catalog support example on the adhoc editor, also showing autocompletion.

Catalog support on the Admin Dashboard, this is howing a Glue metastore with options to enable/disable the catalog showing on the UI