Skip to content

PythonDialect.md

Latest commit

 

History

History
197 lines (140 loc) · 6.25 KB

PythonDialect.md

File metadata and controls

197 lines (140 loc) · 6.25 KB
Raw

The Python Dialect

The Python dialect is an auto-generated Strata DDM dialect that mirrors Python's ast module. Each Python AST node type becomes a DDM operator, and each AST base class (e.g., expr, stmt, keyword) becomes a syntactic category. The top-level ast.mod category is mapped to Init.Command rather than getting its own category, so parsed Python modules appear as Strata commands. This allows Python source files to be represented as Strata programs for downstream analysis.

For general DDM concepts (dialects, operators, syntactic categories, the Init dialect), see the DDM Manual.

Overview

The Python dialect is generated by reflecting over CPython's ast module at runtime. Because the AST structure can change between Python versions, the dialect is version-specific. The Strata toolchain assumes the dialect is generated with CPython 3.14.

The dialect defines:

  • Syntactic categories for each AST base class (expr, stmt, arguments, keyword, etc.) and for special types (int, constant, opt_expr).
  • Operators for each concrete AST node (FunctionDef, BinOp, Call, If, etc.) targeting the appropriate category.
  • Constant operators for Python literals: ConTrue, ConFalse, ConPos, ConNeg, ConString, ConFloat, ConComplex, ConNone, ConEllipsis, ConBytes.
  • Integer operators: IntPos and IntNeg for signed integers.

Operator arguments correspond to the _fields of the Python AST node, with types mapped as follows:

Python AST type DDM category
expr Python.expr
stmt Python.stmt
str Init.Str
int Python.int
object (constants) Python.constant
list[X] Init.Seq(X)
X | None Init.Option(X)

A few Python field names conflict with DDM reserved words and are renamed: op becomes mk_op and type becomes mk_type.

Installation

Install these two packages (. refers to the repo root):

pip install <StrataDDM-repo>/Python/strata
pip install ./Python/strata-python

This requires Python 3.11 or later and installs the amazon.ion dependency.

Command-Line Interface

The strata_python.gen module provides a CLI accessible via python -m strata_python.gen.

Generating the dialect

python -m strata_python.gen dialect <output_dir>

Requires Python 3.13 or later. Writes <output_dir>/Python.dialect.st.ion in binary Ion format. The generated file can be verified with the Strata CLI:

strata check "<output_dir>/Python.dialect.st.ion"

Parsing Python to Strata

python -m strata_python.gen py_to_strata [--dialect <path>] <input.py> <output.st.ion>

Translates a Python source file into a Strata program in Ion format. The --dialect flag specifies a pre-generated dialect file; if omitted and Python >= 3.13, the dialect is generated on the fly.

Example:

python -m strata_python.gen py_to_strata --dialect dialects/Python.dialect.st.ion \
    my_script.py my_script.py.st.ion

The output can be verified with:

strata check --include dialects my_script.py.st.ion

Exit code 100 indicates a Python parse failure.

Checking the AST parser

python -m strata_python.gen check_ast [--dialect <path>] <dir>

Batch-parses all .py files under <dir> and reports how many were successfully converted (prints "Analyzed X of Y files."). Useful for validating the parser against a corpus.

Common flags

  • -q / --quiet — suppress the warning about the amazon.ion C extension not being available.

Python API

The strata.pythonast module exposes the dialect and parser programmatically.

gen_dialect()

from strata.pythonast import gen_dialect

dialect = gen_dialect()

Returns a Dialect object representing the Python dialect. Requires Python 3.13 or later (uses _field_types on AST nodes).

Parser

from strata.pythonast import Parser

parser = Parser(dialect)

Construct a Parser from a Python Dialect (either from gen_dialect() or loaded from an Ion file with Dialect.from_ion()). The constructor validates that the dialect matches the current Python version's AST via check_ast.

parse_module(source, filename)

mapping, program = parser.parse_module(source, filename)
  • source — Python source code as bytes.
  • filename — file path (used in error messages).
  • Returns a (FileMapping, Program) tuple.

The FileMapping can be used to convert byte offsets in SourceRange annotations back to line/column positions.

ast_to_op(mapping, ast_node)

Converts a single Python AST node to a Strata Operation. Called internally by parse_module, but available for working with individual AST nodes.

Full example

Requires Python 3.13+ and the strata package installed (pip install .).

from pathlib import Path
from strata.pythonast import gen_dialect, Parser
import amazon.ion.simpleion as ion

# Generate dialect (Python 3.13+)
dialect = gen_dialect()

# Parse a Python file
source = Path("example.py").read_bytes()
parser = Parser(dialect)
mapping, program = parser.parse_module(source, "example.py")

# Serialize to Ion
with open("example.py.st.ion", "wb") as f:
    ion.dump(program.to_ion(), f, binary=True)

Python Version Compatibility

Python version Dialect generation Parsing with pre-generated dialect
3.11 No Yes (some fields missing*)
3.12 No Yes (some fields missing*)
3.13+ Yes Yes
3.14 Yes (recommended) Yes

* When parsing with an older Python version than the one used to generate the dialect, fields that don't exist in the older AST (e.g., type_params added in 3.12, default_value added in 3.13) are filled with default values (Option fields get None, Seq fields get an empty sequence).