Support TiDB NullEQ pushdown

[windtalker]

What problem does this PR solve?

Issue Number: close #5102

Problem Summary:

TiDB can push down NullEQ expressions, but TiFlash does not fully recognize and execute the pushed-down NullEQ semantics yet, including rough-set filtering in DeltaMerge.

What is changed and how it works?

support nulleq in tiflash
DeltaMerge: fix rough-set semantics for NullEQ

This PR adds TiDB NullEQ function support in TiFlash, wires NullEQ pushdown handling, adds execution tests, and fixes DeltaMerge rough-set/min-max index semantics for NullEQ.

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)
• No code

Side effects

• Performance regression: Consumes more CPU
• Performance regression: Consumes more Memory
• Breaking backward compatibility

Documentation

• Affects user behaviors
• Contains syntax changes
• Contains variable changes
• Contains experimental features
• Changes MySQL compatibility

Release note

Support TiDB NullEQ pushdown in TiFlash.

Summary by CodeRabbit

• New Features

  • Added TiDB null-safe equality (<=>) end-to-end: expression recognition, new tidbNullEQ function semantics, and DeltaMerge filtering via null_equal.
  • Extended nullable-aware behavior across supported types (including string and vectors) with consistent NULL <=> NULL handling.

• Bug Fixes

  • Improved rough filtering and min/max index evaluation for nullable null-safe equality, including correct outcomes for mixed NULL/non-NULL packs.

• Tests

  • Added comprehensive function and storage-layer tests for tidbNullEQ and null_equal, including parsing and negation cases, plus collator forwarding.

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign solotzg for approval. For more information see the Code Review Process.
Please ensure that each of them provides their approval before proceeding.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: edc83dcd-3ea6-4988-a8f2-abfd0e0a39a5

📥 Commits

Reviewing files that changed from the base of the PR and between e53c6b9 and 31474b8.

📒 Files selected for processing (3)

• dbms/src/Storages/DeltaMerge/Filter/NullEqual.h
• dbms/src/Storages/DeltaMerge/Index/tests/gtest_dm_minmax_index.cpp
• dbms/src/Storages/DeltaMerge/tests/gtest_dm_filter_parser_nulleq.cpp

🚧 Files skipped from review as they are similar to previous changes (2)

• dbms/src/Storages/DeltaMerge/tests/gtest_dm_filter_parser_nulleq.cpp
• dbms/src/Storages/DeltaMerge/Index/tests/gtest_dm_minmax_index.cpp

---

📝 Walkthrough

Walkthrough

This PR adds TiDB NullEQ (<=>) support across coprocessor function mapping, function execution, DeltaMerge filter parsing, and MinMaxIndex rough-checking, with new unit and integration coverage for nullable and non-nullable cases.

Changes

NullEQ Push-Down Implementation

Layer / File(s)	Summary
DAGUtils NullEQ signature mapping dbms/src/Flash/Coprocessor/DAGUtils.cpp, dbms/src/Flash/Coprocessor/tests/gtest_tidb_null_eq_func.cpp	Maps TiDB NullEQ* scalar signatures to tidbNullEQ and verifies the mapping in a new gtest.
tidbNullEQ function implementation and tests dbms/src/Functions/FunctionsComparison.cpp, dbms/src/Functions/tests/gtest_tidb_null_eq.cpp	Adds FunctionTiDBNullEQ, registers it, and tests NULL-safe equality behavior across nullable inputs, constants, and collators.
NullEqual RS operator and parser dispatch dbms/src/Storages/DeltaMerge/Filter/NullEqual.h, dbms/src/Storages/DeltaMerge/Filter/RSOperator.{cpp,h}, dbms/src/Storages/DeltaMerge/FilterParser/FilterParser.{cpp,h}, dbms/src/Storages/DeltaMerge/tests/gtest_dm_filter_parser_nulleq.cpp	Introduces the NullEqual rough-set operator, adds parser routing for NullEQ* signatures, and covers the lowering behavior in parser tests.
MinMaxIndex checkNullEqual implementation dbms/src/Storages/DeltaMerge/Index/MinMaxIndex.{cpp,h}, dbms/src/Storages/DeltaMerge/Index/tests/gtest_dm_minmax_index.cpp	Adds nullable null-equality rough-check support in MinMaxIndex, updates string boundary handling, and extends index tests for NullEQ and related parsing cases.

Estimated code review effort: 4 (Complex) | ~60 minutes

Sequence Diagram(s)

sequenceDiagram
  participant TiDB Expr
  participant FilterParser
  participant NullEqual
  participant MinMaxIndex

  TiDB Expr->>FilterParser: NullEQ* expression
  FilterParser->>FilterParser: map to RSFilterType::NullEqual
  FilterParser->>NullEqual: createNullEqual(attr, value)
  NullEqual->>MinMaxIndex: checkNullEqual(...)
  MinMaxIndex-->>NullEqual: RSResults
  NullEqual-->>FilterParser: roughCheck result

Loading

Possibly related PRs

• pingcap/tiflash#10769: Overlaps with the same null_equal DeltaMerge pipeline, including parser routing and MinMaxIndex::checkNullEqual.

Suggested labels: approved, lgtm

Suggested reviewers: JaySon-Huang, gengliqi

Poem

I hop through bytes on silent paws,
And <=> now knows the NULL-safe laws.
Min-max checks and parsers sing,
A tidy tidbNullEQ spring!
🐇

🚥 Pre-merge checks | ✅ 3 | ❌ 2

❌ Failed checks (2 warnings)

Check name	Status	Explanation	Resolution
Linked Issues check	⚠️ Warning	Issue #5102 requires at least one integration test, but the PR only adds unit tests and no integration test is shown.	Add at least one integration test covering NullEq pushdown to satisfy #5102, alongside the existing unit tests.
Docstring Coverage	⚠️ Warning	Docstring coverage is 3.33% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (3 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly and concisely summarizes the main change: TiDB NullEQ pushdown support.
Description check	✅ Passed	The description follows the template with issue number, problem summary, change summary, tests, and release note.
Out of Scope Changes check	✅ Passed	The code changes all support NullEQ pushdown behavior, parsing, execution, and DeltaMerge semantics.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@dbms/src/Storages/DeltaMerge/Index/tests/gtest_dm_minmax_index.cpp`:
- Around line 2448-2455: The DAGQueryInfo construction in
gtest_dm_minmax_index.cpp is using the wrong overload and is missing required
parameters. Update the std::make_unique<DAGQueryInfo> call in the minmax index
test to match the full constructor signature used by ParseIn, including the
missing fts_query_info and used_indexes arguments in the correct order. Use the
DAGQueryInfo constructor and the surrounding ParseIn-style argument sequence as
the reference for locating and fixing this call.

In `@dbms/src/Storages/DeltaMerge/tests/gtest_dm_filter_parser_nulleq.cpp`:
- Around line 95-103: The DAGQueryInfo construction in the null-eq parser test
is missing required arguments, so update the std::make_unique<DAGQueryInfo> call
to pass both fts_query_info and empty_used_indexes in the correct position. Use
the existing local values and match the constructor signature in DAGQueryInfo
exactly so the test builds against the current API.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: fcd47c8c-fa93-4f5a-8ca5-a5a90c7d977b

📥 Commits

Reviewing files that changed from the base of the PR and between 63cff9a and e53c6b9.

📒 Files selected for processing (13)

• dbms/src/Flash/Coprocessor/DAGUtils.cpp
• dbms/src/Flash/Coprocessor/tests/gtest_tidb_null_eq_func.cpp
• dbms/src/Functions/FunctionsComparison.cpp
• dbms/src/Functions/tests/gtest_tidb_null_eq.cpp
• dbms/src/Storages/DeltaMerge/Filter/NullEqual.h
• dbms/src/Storages/DeltaMerge/Filter/RSOperator.cpp
• dbms/src/Storages/DeltaMerge/Filter/RSOperator.h
• dbms/src/Storages/DeltaMerge/FilterParser/FilterParser.cpp
• dbms/src/Storages/DeltaMerge/FilterParser/FilterParser.h
• dbms/src/Storages/DeltaMerge/Index/MinMaxIndex.cpp
• dbms/src/Storages/DeltaMerge/Index/MinMaxIndex.h
• dbms/src/Storages/DeltaMerge/Index/tests/gtest_dm_minmax_index.cpp
• dbms/src/Storages/DeltaMerge/tests/gtest_dm_filter_parser_nulleq.cpp

[ti-chi-bot]

@windtalker: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-sanitizer-tsan	31474b8	link	false	/test pull-sanitizer-tsan
pull-sanitizer-asan	31474b8	link	false	/test pull-sanitizer-asan
pull-unit-next-gen	31474b8	link	true	/test pull-unit-next-gen
pull-unit-test	31474b8	link	true	/test pull-unit-test

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.