A micro-library of stream components for building custom JSON and JSONC processing pipelines with a minimal memory footprint — parse, filter, and transform JSON far larger than available memory with a SAX-inspired token API, on Node.js or Web Streams.

Optimised tokenizer/lexer generator! 🐄 Uses /y for performance. Moo.

🌿 NodeJS PHP Parser - extract AST or tokens

Tiny JavaScript tokenizer.

JS tokenizer for LLaMA 1 and 2

Collection of developer toolkits

bison / YACC / LEX in JavaScript (LALR(1), SLR(1), etc. lexer/parser generator)

JS tokenizer for LLaMA 3 and LLaMA 3.1

😻 Markdown with easy tokenization, a fast highlighter, and a lean HTML sanitizer

Tokenizes Chinese texts into words.

JavaScript Reverse Engineering Toolkit (JSRETK) - Experimental tools for analyzing (minified/obfuscated) JavaScript

Multilingual tokenizer that automatically tags each token with its type

A small ECMAScript parser, tokenizer and minifier written in JavaScript.

A very small and very fast spec compliant css lexer

A modern CSS value compiler in JavaScript

A simple interpreter for a large subset of Pascal language written for educational purposes

Compress GraphQL Query String

POS Tagger, lemmatizer and stemmer for french language in javascript

a lightweight no-dependency fork from transformers.js (only tokenizers)

The first language I made.