Skip to content

Fix MySQL DELETE using CONCAT_WS preventing index usage#5756

Open
hashwnath wants to merge 2 commits into
SQLMesh:mainfrom
hashwnath:fix/5711-mysql-delete-concat-ws
Open

Fix MySQL DELETE using CONCAT_WS preventing index usage#5756
hashwnath wants to merge 2 commits into
SQLMesh:mainfrom
hashwnath:fix/5711-mysql-delete-concat-ws

Conversation

@hashwnath

Copy link
Copy Markdown

Summary

Fixes #5711

  • Override _replace_by_key in MySQLEngineAdapter to use a JOIN-based DELETE for composite keys instead of CONCAT_WS('__SQLMESH_DELIM__', ...) which prevents MySQL/MariaDB from using indexes
  • Single-key DELETEs continue to use the existing efficient IN-based approach since indexes work fine for single columns
  • Add tests verifying both composite key (JOIN-based) and single key (IN-based) DELETE behavior

Problem

When using INCREMENTAL_BY_UNIQUE_KEY with MySQL/MariaDB and composite keys, SQLMesh generates:

DELETE FROM `target_table`
WHERE CONCAT_WS('__SQLMESH_DELIM__', `col1`, `col2`)
IN (SELECT CONCAT_WS('__SQLMESH_DELIM__', `col1`, `col2`) FROM `temp_table`)

MySQL cannot use indexes on CONCAT_WS() expressions, causing full table scans that lead to multi-hour query times and connection timeouts on larger tables.

Fix

For composite keys, generate a JOIN-based DELETE that allows MySQL to use existing indexes:

DELETE `_target`
FROM `target_table` AS `_target`
INNER JOIN `temp_table` AS `_temp`
ON `_target`.`col1` = `_temp`.`col1`
AND `_target`.`col2` = `_temp`.`col2`

This changes the EXPLAIN plan from type=ALL (full scan) to type=ref (index lookup), reducing query times from hours to seconds.

Test plan

  • New test test_replace_by_key_composite_uses_join_delete verifies JOIN-based DELETE for composite keys
  • New test test_replace_by_key_single_key_uses_in verifies single-key DELETEs still use the IN-based approach
  • Existing MySQL adapter tests pass
  • Existing mixin tests pass
  • Existing base engine adapter tests pass
@StuffbyYuki

Copy link
Copy Markdown
Collaborator

@hashwnath Thanks for the fix!

A few things before merge:

  • Do you think it's worth adding a test with a 3-column composite key (matching the issue scenario)? And a test for expression-based composite keys (e.g. DATE_TRUNC + column), since insert_overwrite_by_partition also hits _replace_by_key?

Let me know if I'm missing anything!

Override _replace_by_key in MySQLEngineAdapter to use a JOIN-based
DELETE for composite keys instead of CONCAT_WS('__SQLMESH_DELIM__', ...).

MySQL/MariaDB cannot use indexes on CONCAT_WS expressions, causing full
table scans on every incremental run. The JOIN-based approach allows the
engine to use existing composite indexes, reducing query times from hours
to seconds on large tables.

Single-key DELETEs continue to use the efficient IN-based approach.
@StuffbyYuki StuffbyYuki force-pushed the fix/5711-mysql-delete-concat-ws branch from ee0d9f8 to a6907c0 Compare May 31, 2026 06:24
@StuffbyYuki

Copy link
Copy Markdown
Collaborator

@hashwnath Following up on my last comment

Fix QueryOrDF import path, replace reduce with exp.and_ for type
safety, add 3-column composite key test and expression-based
composite key test via insert_overwrite_by_partition.

Signed-off-by: Hashwanth S <s.hashwanth531@gmail.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Hashwanth S <s.hashwanth531@gmail.com>
@hashwnath

Copy link
Copy Markdown
Author

Hi, added the 3-col composite key test and expression-based key test, also fixed the mypy/formatting issues

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

2 participants