Grammar and editor tooling reference

This page is the exact map for FCSTM grammar, syntax highlighting, and editor maintenance. For the task workflow, see Grammar and editor tasks. For rationale, see Grammar tooling explanation.

Canonical source and generated files

Grammar and editor files

Area	Path	Source or generated	Notes
Parser grammar	pyfcstm/dsl/grammar/GrammarParser.g4	source	Parser rules for FCSTM constructs.
Lexer grammar	pyfcstm/dsl/grammar/GrammarLexer.g4	source	Tokens, keywords, literals, and operators.
Python parser output	pyfcstm/dsl/grammar/	generated	Regenerated by make antlr_build; do not hand-edit generated parser files.
DSL listener	pyfcstm/dsl/listener.py	source	Converts parse events into AST nodes.
DSL AST nodes	pyfcstm/dsl/node.py	source	Syntax tree dataclasses and export helpers.
Model importer	pyfcstm/model/imports.py and related pyfcstm/model/ modules	source	Converts AST nodes into semantic model objects and diagnostics.
Pygments lexer	pyfcstm/highlight/pygments_lexer.py	source	Documentation and Python-side syntax highlighting.
TextMate grammar	editors/fcstm.tmLanguage.json	source	Shared TextMate-compatible highlighting grammar.
Editor validation	editors/validate.py	source command	Repository validation for highlighter/editor asset consistency.
JavaScript frontend	editors/jsfcstm/	source and generated local assets	JavaScript parser/runtime assets for editor integrations.
VSCode extension	editors/vscode/	source and build output	VSCode packaging and authoring features.
LLM grammar guide	pyfcstm/llm/fcstm_grammar_guide.md	source prompt asset	Update with syntax changes and refresh .sha256 with make sha256.

Core maintenance commands

Commands

Command	Purpose	Typical trigger
make antlr	Download/setup ANTLR support when needed.	First local grammar-maintenance setup.
make antlr_build	Regenerate parser outputs after editing GrammarParser.g4 or GrammarLexer.g4.	Any grammar file change.
python editors/validate.py	Validate syntax highlighting and editor asset consistency.	Grammar, keyword, operator, Pygments, or TextMate changes.
make vscode	Build the VSCode extension package.	VSCode package or extension integration changes.
make vscode_clean	Remove VSCode extension build artifacts.	Local cleanup.
make sha256	Refresh generated checksum sidecars.	LLM grammar guide content change.
SKIP_SLOW_TESTS=1 make unittest RANGE_DIR=./llm	Verify LLM guide packaging and checksum behavior.	LLM guide or prompt-asset changes.

Pygments and Sphinx facts

The package registers pyfcstm.highlight.pygments_lexer.FcstmLexer through the pygments.lexers entry point in setup.py. The canonical alias is fcstm; fcsm is also accepted by the lexer.

Programmatic load check:

from pygments.lexers import get_lexer_by_name

lexer = get_lexer_by_name("fcstm")

Sphinx also registers the lexer in docs/source/conf.py. User-facing FCSTM examples should use:

.. code-block:: fcstm

   state Root {
       state Idle;
       [*] -> Idle;
   }

TextMate facts

The repository TextMate grammar is editors/fcstm.tmLanguage.json. It is the source for TextMate-compatible highlighting and is copied into the VSCode extension’s syntaxes/ area during packaging.

For Sublime Text integration, place the same file under a package directory such as FCSTM in Preferences -> Browse Packages.

VSCode extension facts

VSCode file map

File or directory	Purpose
editors/vscode/package.json	Extension manifest, commands, activation, language contribution, and scripts.
editors/vscode/language-configuration.json	Comment, bracket, indentation, and editor language behavior.
editors/vscode/src/diagnostics.ts	Problems-panel diagnostics and inline squiggles.
editors/vscode/src/symbols.ts	Document symbols for Outline and breadcrumbs.
editors/vscode/src/completion.ts	IntelliSense for keywords, constants, built-ins, and document-local symbols.
editors/vscode/src/hover.ts	Contextual help for event scopes, pseudo states, lifecycle keywords, and aspect syntax.
editors/vscode/snippets/fcstm.code-snippets	Snippets for common variable, state, transition, and lifecycle forms.
editors/vscode/dist/	Bundled output.
editors/vscode/build/	Local .vsix package output.

Local VSIX installation:

code --install-extension editors/vscode/build/fcstm-language-support-0.1.0.vsix

VSCode verification suites

Focused suites

Command	Focus
make verify-p0.2	Parser integration.
make verify-p0.3	Syntax diagnostics.
make verify-p0.4	Document symbols.
make verify-p0.5	Completion support.
make verify-p0.6	Hover documentation.
make verify	Aggregate extension verification, including newer semantic, import, preview, and end-to-end checks.

Operator ordering facts

Lexer patterns and highlighters should put multi-character operators before single-character operators.

Operator ordering examples

Longer token	Must precede	Why
**	*	Exponentiation must not be tokenized as two multiplication tokens.
<<	<	Shift must not split into two less-than tokens.
<= / >=	< / >	Comparisons must keep their equality suffix.
== / !=	= / !	Equality and inequality must not split into assignment or negation tokens.
&& / ||	& / |	Logical operators must not split into bitwise-looking pieces.

Keyword update checklist

When syntax changes add a keyword or operator, update these facts together:

1. ANTLR grammar files under pyfcstm/dsl/grammar/.

2. Parser outputs via make antlr_build.

3. Listener and AST handling when parse-tree shape changes.

4. Model import and validation when semantic meaning changes.

5. Pygments lexer groups.

6. TextMate grammar keyword/operator patterns.

7. Editor validation expectations.

8. VSCode diagnostics, completions, hovers, and snippets when authoring behavior changes.

9. DSL guide, examples, and tests when user-facing syntax changes.

10. LLM grammar guide and checksum when prompt-facing syntax changes.

Do not document generated parser paths as the source of truth. The .g4 files are the source; generated parser files are outputs.