Add gdal mdim get-refs algorithm by mdsumner · Pull Request #14677 · OSGeo/gdal

mdsumner · 2026-05-28T00:51:26Z

gdal mdim get-refs command.

Creates a vector layer (no geometry) that lists the path, offset, size and info of chunk byte references from a multidimensional raster array.

The emitted table also includes dim_0, ...dim_n fields that identify the chunk's address (and allows filter queries), and present a boolean for whether the chunk exists.

Metadata is also recorded in the layer for ARRAY_NAME, DTYPE, DIM_0{:N}, and .papszInfo items in the form CODEC_%s eg. CODEC_COMPRESSION, CODEC_FILTER.

AI tool usage

AI Claude assisted

Tasklist

Description

Usage: gdal mdim get-refs [OPTIONS] <INPUT> <OUTPUT>

Return byte references from a multidimensional raster source as vector/table layer.

Positional arguments:
  -i, --dataset, --input <INPUT>                       Input multidimensional raster dataset [required]
  -o, --output <OUTPUT>                                Output vector dataset [required]

Common Options:
  -h, --help                                           Display help message and exit
  --json-usage                                         Display usage as JSON document and exit
  --config <KEY>=<VALUE>                               Configuration option [may be repeated]
  -q, --quiet                                          Quiet mode (no progress bar)

Options:
  -f, --of, --format, --output-format <OUTPUT-FORMAT>  Output format [required]
  --array <ARRAY>                                      Name of the array, used to restrict the output to the specified array. [required]
  --overwrite                                          Whether overwriting existing output is allowed
  --co, --creation-option <KEY>=<VALUE>                Creation option [may be repeated]

Advanced Options:
  --oo, --open-option <KEY>=<VALUE>                    Open options [may be repeated]
  --if, --input-format <INPUT-FORMAT>                  Input formats [may be repeated]

For more details, consult https://gdal.org/programs/gdal_mdim_get_refs.html

Example

gdal mdim get-refs --array /Band1 autotest/gdrivers/data/netcdf/byte_chunked_multiple.nc \
    /tmp/Band1.parquet --of Parquet

gdal vector info /tmp/Band1.parquet --layer Band1 --features
INFO: Open of `/tmp/Band1.parquet'
      using driver `Parquet' successful.

Layer name: Band1
Metadata:
  ARRAY_NAME=/Band1
  DTYPE=Byte
  DIM_0_NAME=y
  DIM_0_SIZE=20
  DIM_0_BLOCK=10
  DIM_0_CHUNKS=2
  DIM_1_NAME=x
  DIM_1_SIZE=20
  DIM_1_BLOCK=10
  DIM_1_CHUNKS=2
  CODEC_COMPRESSION=DEFLATE
  CODEC_FILTER=SHUFFLE
Geometry: None
Feature Count: 4
Layer SRS WKT:
(unknown)
dim_0: Integer64 (0.0)
dim_1: Integer64 (0.0)
present: Integer(Boolean) (0.0)
path: String (0.0)
offset: Integer64 (0.0)
size: Integer64 (0.0)
info: String (0.0)
OGRFeature(Band1):0
  dim_0 (Integer64) = 0
  dim_1 (Integer64) = 0
  present (Integer(Boolean)) = 1
  path (String) = autotest/gdrivers/data/netcdf/byte_chunked_multiple.nc
  offset (Integer64) = 6108
  size (Integer64) = 86
  info (String) = COMPRESSION=DEFLATE; FILTER=SHUFFLE

OGRFeature(Band1):1
  dim_0 (Integer64) = 0
  dim_1 (Integer64) = 1
  present (Integer(Boolean)) = 1
  path (String) = autotest/gdrivers/data/netcdf/byte_chunked_multiple.nc
  offset (Integer64) = 6194
  size (Integer64) = 74
  info (String) = COMPRESSION=DEFLATE; FILTER=SHUFFLE

OGRFeature(Band1):2
  dim_0 (Integer64) = 1
  dim_1 (Integer64) = 0
  present (Integer(Boolean)) = 1
  path (String) = autotest/gdrivers/data/netcdf/byte_chunked_multiple.nc
  offset (Integer64) = 5959
  size (Integer64) = 74
  info (String) = COMPRESSION=DEFLATE; FILTER=SHUFFLE

OGRFeature(Band1):3
  dim_0 (Integer64) = 1
  dim_1 (Integer64) = 1
  present (Integer(Boolean)) = 1
  path (String) = autotest/gdrivers/data/netcdf/byte_chunked_multiple.nc
  offset (Integer64) = 6033
  size (Integer64) = 75
  info (String) = COMPRESSION=DEFLATE; FILTER=SHUFFLE

rouault

not a bad idea, but clearly this doesn't follow our LLM tool policy. The code has clearly not been read carefully. A lot of comments are hyper verbose, with typical "LLM plan garbage" references. I feel some of the tests are also a bit more verbose than what would be strictly needed (although it might just be my aversion for being exposed to prior verbosity)

rouault · 2026-05-28T01:04:44Z

+/******************************************************************************
+ *
+ * Project:  GDAL
+ * Purpose:  gdal "mdim get-refs" subcommand


get-refs is a bit mysterious until you read the doc.
brainstorming: get-chunk-location, chunk-loc

or chunk-info ?

list-chunks or extract-chunks - I don't have a really strong feel here (get-chunk-location is absolutely fine so I'll go with that unless a stronger candidate or position arrives)

rouault · 2026-05-28T01:05:36Z

+}  // namespace
+
+/************************************************************************/
+/*                        GDALMdimGetRefsAlgorithm()                  */


formatting issue. your pre-commit is not setup ?

it is setup, I will check

rouault · 2026-05-28T01:07:35Z

+bool GDALMdimGetRefsAlgorithm::RunImpl(GDALProgressFunc pfnProgress,
+                                       void *pProgressData)
+{
+    // ----------------------------------------------------------------------


all those "stage", "A1", "D6" etc are really LLM noise. Please remove in general.

And you can pretty much remove that particular 5-line comment for a 2 line effective code...

rouault · 2026-05-28T01:07:57Z

+    auto poSrcDS = m_inputDataset.GetDatasetRef();
+    CPLAssert(poSrcDS);
+
+    // A2. GetRootGroup(). Null root => driver lacks mdim support => fail with a


make comment more succinct (remove the ==> fail part .... that's obvious from the code)

rouault · 2026-05-28T01:08:41Z

+    auto poRootGroup = poSrcDS->GetRootGroup();
+    CPLDebug("MDIM-GET-REFS", "input: %s, root group: %s",
+             poSrcDS->GetDescription(), poRootGroup ? "present" : "NULL");
+


Suggested change

rouault · 2026-05-28T01:49:06Z

+
+* ``ARRAY_NAME`` — the fullname of the input array
+* ``DTYPE`` — the array's data type (e.g. ``Int16``, ``Float32``)
+* ``DIM_N_NAME``, ``DIM_N_SIZE``, ``DIM_N_BLOCK``, ``DIM_N_CHUNKS`` — for


make a global pass to replace UTF-8 dash by ASCII dash

rouault · 2026-05-28T01:49:28Z

+* Only a single array can be emitted (``--array`` is required).
+* Arrays without natural block size decline with a ``not chunk-enumerable`` error.
+* The inline data payload is not extracted as a binary field.
+* No geometry column is emitted.


Suggested change

* No geometry column is emitted.

rouault · 2026-05-28T01:50:40Z

+-----------
+
+* Only a single array can be emitted (``--array`` is required).
+* Arrays without natural block size decline with a ``not chunk-enumerable`` error.


not really a limitation IMHO (in the sense there's nothing we can do about that)

Suggested change

* Arrays without natural block size decline with a ``not chunk-enumerable`` error.

rouault · 2026-05-28T01:51:46Z

+
+   .. code-block:: bash
+
+       ogrinfo ocean_chunks.parquet -sql \


Suggested change

ogrinfo ocean_chunks.parquet -sql \

gdal vector info ocean_chunks.parquet --features --sql \

rouault · 2026-05-28T01:57:33Z

+
+
+@pytest.mark.require_driver("Parquet")
+def test_gdalalg_mdim_get_refs_parquet_boolean_subtype(tmp_vsimem):


does it exercice any code path not already exercised ? I don't think so

github-actions · 2026-06-26T03:44:55Z

The GDAL project highly values your contribution and would love to see this work merged! Unfortunately this PR has not had any activity in the last 28 days and is being automatically marked as "stale". If you think this pull request should be merged, please check

that all unit tests are passing
that all comments by reviewers have been addressed
that there is enough information for reviewers, in particular link
to any issues which this pull request fixes
that you have written unit tests where possible
In case you should have any uncertainty, please leave a comment and we will be happy to help you proceed with this pull request.
If there is no further activity on this pull request, it will be closed in 2 weeks.

Add gdal mdim get-refs algorithm

aa69a6e

mdsumner marked this pull request as draft May 28, 2026 00:51

rouault reviewed May 28, 2026

View reviewed changes

apply review changes (some)

a68f158

github-actions Bot added the stale label Jun 26, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Add gdal mdim get-refs algorithm#14677

Add gdal mdim get-refs algorithm#14677
mdsumner wants to merge 2 commits into
OSGeo:masterfrom
mdsumner:feature/mdim-get-refs

mdsumner commented May 28, 2026 •

edited

Loading

rouault left a comment

rouault May 28, 2026

rouault May 28, 2026

mdsumner May 28, 2026

rouault May 28, 2026

mdsumner May 28, 2026

rouault May 28, 2026

rouault May 28, 2026

rouault May 28, 2026

rouault May 28, 2026

rouault May 28, 2026

rouault May 28, 2026

rouault May 28, 2026

rouault May 28, 2026

github-actions Bot commented Jun 26, 2026

Labels

2 participants

	ogrinfo ocean_chunks.parquet -sql \
	gdal vector info ocean_chunks.parquet --features --sql \



		@pytest.mark.require_driver("Parquet")
		def test_gdalalg_mdim_get_refs_parquet_boolean_subtype(tmp_vsimem):

Uh oh!

Uh oh!

Conversation

mdsumner commented May 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

AI tool usage

Tasklist

Description

Example

rouault left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions Bot commented Jun 26, 2026

Labels

2 participants

mdsumner commented May 28, 2026 •

edited

Loading