roots-parse: the JavaScript / TypeScript parser

1. API #

Every function takes the source as a string. Offsets are character indices (code points) into that string. Every END is exclusive. Errors are signalled as js-syntax-error (section 5).

Parsing #

Type stripping #

(strip-types source &key filename) → JavaScript string.

• Erasable TypeScript (annotations, interfaces, type aliases, generics, as, satisfies, !, declare …, overloads, abstract, modifiers, ?, !:, this parameters, implements, import type, type-only specifiers, type-only namespaces, index signatures …) is replaced by whitespace. Positions are kept, line terminators are kept, and replacement characters keep both the UTF-8 and the UTF-16 length (section 3).
• enum (including const enum) is transformed the way Node's transform mode does it, and so are constructor parameter properties. Only the lines they are on change.
• Constructs that cannot be stripped are errors. Each error explains the rewrite: a namespace with runtime code (C120), import x = … (C121), export = (C122), <T>x assertions (C123), parameter decorators (C124).
• JavaScript input with no TypeScript syntax is returned unchanged (eq to the argument).

Module analysis #

(module-imports source &key typescript) → list in source order of

(:specifier "./x.vue" :start s :end e :kind :static|:dynamic|:export-from
 :type-only bool :attributes (("type" . "json")))

• :start/:end cover the string literal including its quotes. So (replace-range source start end "\"./x.js\"") rewrites the specifier in place.
• :static covers every import declaration, side-effect imports included.
• :export-from covers export … from and export * from.
• :dynamic covers import("…") when the argument is a string literal or a template without substitutions. Computed dynamic imports are not listed.
• :type-only is true for import type …, export type … from and export type * from, which strip-types erases. An import { type A } from "m" stays a runtime import (import {} from "m", as with verbatimModuleSyntax).

(module-bindings source &key typescript) → the top-level runtime bindings in source order, each

(:name "count" :kind :const|:let|:var|:function|:class|:import|:enum
 :exported bool :start s :end e        ; START/END of the binding identifier
 :source "./mod" :imported "x"|"default"|"*")   ; imports only

• Destructuring declarations contribute every name they bind.
• using declarations count as :const.
• Overloads collapse into one :function.
• Type aliases, interfaces, type-only imports and specifiers, and declared names are not bindings.
• This list is what a <script setup> returns.

Template expressions (scope analysis) #

(rewrite-expression source &key locals (prefix "_ctx.") typescript statement refs only)
  → (values js-code prefixed-names)

Every free identifier reference (one not bound inside source) is rewritten as follows:

With :only (names…), only those names get the prefix and nothing else changes (e.g. destructured props: title → __props.title).

Identifiers are left alone in all of these positions:

• names bound inside the expression: arrow and function parameters, destructuring, let/const/class/function declarations in statement mode, catch parameters;
• non-computed member names (a.b changes only a);
• object keys, method names, labels, import.meta, new.target;
• TypeScript type positions.

Assignments and updates are rewritten like any other reference (count++ → _ctx.count++). With :typescript t the source is TypeScript and its types are stripped. With :statement t the source is a statement list such as count++; save($event) or a <script setup> body. It is parsed like a module body inside an async function, so return, await and import/export declarations are accepted; imported names count as bound. The second value lists the names that received prefix, in order of first appearance.

(free-identifiers source &key typescript statement) → the names source references without binding them, globals included, in order of first appearance.

(binding-names source &key typescript) → the names a parameter list or pattern binds. It is meant for v-for aliases and slot props: "(item, index)" → ("item" "index"), "{ id, name: n }" → ("id" "n").

AST access #

• node, node-p, node-type, node-start, node-end, node-props, and (node-get node key) (also settable).
• (node-source node source) returns the text a node covers.
• (walk-ast fn node) visits every node depth-first; if fn returns :skip, the node's children are not visited.
• (position-line-column source offset) → 1-based line and column.
• (replace-range source start end text) replaces a range of the source.

Annotation comments #

Comments are dropped, except that an annotation comment marks the node that starts at the token after it. A comment is an annotation when its text, after an optional leading ! and whitespace, starts with @__PURE__, #__PURE__, @__NO_SIDE_EFFECTS__ or #__NO_SIDE_EFFECTS__ followed by a word boundary (the rule esbuild, rollup and terser share): /*#__PURE__*/, /* @__PURE__ */, /*! #__NO_SIDE_EFFECTS__ */ and the line comment // @__NO_SIDE_EFFECTS__ all count. Only whitespace and other comments may separate the annotation from its token.

Anything else before the token (a parenthesis, an operator, a class) takes no mark; unannotated nodes have no such property at all. Annotations never change the token stream: a multi-line annotation still counts as a line break.

Scope trees (whole modules) #

(analyze-scopes program &optional source) → a scope-tree for a :program node from parse-module or parse-script. scope-tree-root is the module scope; scope-tree-source keeps source when given. Running it again on the same AST replaces the previous result.

A scope (scope-kind :module/:script for the program, :function, :arrow, :block, :for, :catch, :class, :static-block) has scope-parent, scope-children (source order), scope-node (the node that opens it), scope-bindings (declaration order) and, for the module scope, scope-free (references to free identifiers, source order). (scope-lookup scope name) finds a binding in one scope, (scope-resolve scope name) walks outwards.

A binding has binding-name, binding-kind (:var :let :const :function :class :param :catch :import :function-name :class-name :arguments :enum), binding-scope, binding-node (the first binding identifier; NIL for the implicit arguments), binding-nodes (every declaration of the name in that scope: redeclared vars, repeated function declarations) and binding-references. Two more slots serve whoever renames:

• binding-sites: for a var whose declarator is written in a nested scope (if (x) { var v }, catch (e) { var v }, a for body), a list of (identifier . scope) with the scope the token stands in. The binding belongs to the function, the token to the block: a new name has to be free in every scope from the site up to the binding's own. Declarators written in the function's own scope have no site.
• binding-pinned: true for the two bindings of catch (e) { var e = 1 }. ECMAScript (B.3.4) allows a var to redeclare a catch parameter that is a plain identifier; the token e then declares the function's e and assigns the parameter. No renaming of either binding alone keeps the program, so both must keep their names. (V8 accepts the head of a for…of there too, against the standard; it is pinned as well, not rejected.)

Redeclarations. (scope-tree-redeclarations tree) lists every declaration an engine rejects with "Identifier 'x' has already been declared", as (identifier-node . name) in source order, and (check-redeclarations tree &key filename) signals the first as js-syntax-error C110 (the tree must have been built with its source) or returns the tree. Reported: a second declaration of a name that is declared with let, const, class, an import, or a function declaration in a block or at the top level of a module (let a; var a, function f(a) { let a }, two functions of one name in a module); a var written in a block below such a name ({ let b; { var b } }, for (let i;;) { var i }); a lexical declaration in a catch block that repeats the parameter; a repeated catch parameter; a repeated parameter (module code). Not reported: var/var, a parameter and a body var or function, functions repeated in a function body, a static block or a script, the body redeclaring a function expression's own name, the pinned catch case above, and anything TypeScript merges (enums; types, overload signatures and declared names are no bindings). In a script (parse-script) a function repeated in a block and a repeated parameter pass, as sloppy code allows. parse-module alone does not run this check: it needs the scope tree, and callers that only strip types or read imports do not pay for one. The bundler runs it on every module it analyses and the minifier on its own output (docs/bundling.md). The rules were compared with Node 24 (vm.SourceTextModule) on the cases of parse-scope-redeclarations-*, and the corpus test requires that no published file is reported.

A reference has reference-node (the identifier), reference-kind (:read; :write for assignment and update targets; :shorthand for { a } in an object literal or pattern; :export-specifier for the local of export { a as b }), reference-target-p (the identifier is assigned to: always for :write, for a :shorthand in an assignment pattern), reference-scope (where it occurs) and reference-binding (NIL when free).

Helpers: (binding-of tree identifier) → the binding an identifier declares or refers to (NIL when free or not a reference); (reference-of tree identifier) → its reference (NIL for keys, labels, binding identifiers …); (scope-of node) → the scope a node opens (program, function, arrow, class, block, switch, catch clause, static block, a function's body block, a for with a let/const head) or the scope an identifier is declared or used in; (references-of binding); (free-identifiers-of tree) → names in order of first appearance; (free-references-of tree) → their references.

The rules are those of section 4, applied to whole modules (strict code):

• var hoists to the nearest function, arrow, static block or module scope; a function declaration binds where it stands (the function scope for a body-level one, its block otherwise); let/const/class/using (:const) bind in their block; TypeScript enum binds as :enum.
• A function scope holds the parameters, the body's declarations (the body block is the function scope itself) and, for a function expression, its own name (:function-name). arguments is bound implicitly (:arguments, no identifier) in the nearest non-arrow function when it is referenced; at module level it is free. Functions without a body (TypeScript overloads, abstract methods) open no scope. An expression in the parameter list (a default value, a computed key, also inside an arrow or class nested there) sees the parameters, the function's own name and arguments, but not what the body declares: in function f(a = x) { var x } the default reads the outer x, as ECMAScript's separate variable environment for the body has it. A body var that redeclares a parameter stays one binding with it.
• A class declaration's name is one binding, declared in the enclosing scope and visible inside the class body, so a rename edits one token. A class expression's name is bound inside only (:class-name). The heritage, computed keys, field initializers and members are in the class scope; class decorators are outside it; a static block is its own var scope.
• A for statement whose head declares with let/const opens a :for scope that also holds the loop's right-hand side and body; var heads and expression heads open none. A catch clause opens a :catch scope for its parameter, its block a child :block. A switch body is one :block.
• Not references: labels, property keys, non-computed member names, method names, import.meta/new.target, import and export specifier names (except export { local }), export { a } from …, and TypeScript type positions. Binding patterns declare; their defaults and computed keys are references. Assignment patterns reference ([a] = x is a :write, ({ a } = x) a :shorthand with reference-target-p).

The corpus test parse-corpus-scope-trees-agree-with-free-references checks that the tree and free-identifiers (an independent implementation) find the same free references on every sampled module.

CommonJS test #

(commonjs-module-p program &optional tree) → :require, :module, :exports or NIL. True when the parsed module has no import or export declaration (an import() expression does not count) and either calls a free require or references a free module or exports; a bound require (function f(require) {…}), typeof require and strings do not count. tree is the program's scope tree; it is computed when not given. UMD wrappers (typeof module !== 'undefined') test as :module.

Edit plans #

(let ((plan (make-edit-plan :source "aaa bbb ccc")))
  (insert-text plan 0 "<")          ; before the character at 0
  (replace-text plan 4 7 "B")       ; bbb → B
  (erase-range plan 8 11)           ; ccc removed
  (render-edits plan))              ; → "<aaa B "

make-edit-plan, edit-plan-source, insert-text, replace-text, erase-range and render-edits (&key start end renders a part) work on character offsets of the plan's source. Rules: an insertion at P goes before the character at P, so insertions at the start of a replaced range precede it and ones at its end follow it; several insertions at one position keep their order; a replacement nested in another and insertions strictly inside a replaced range are dropped (erasing a statement wins over the edits made inside it); the same range replaced twice: the last edit wins; replacements that overlap only partly signal an error. When rendering a part, an insertion at end belongs to the next range unless end is the end of the source. The type stripper and rewrite-expression are built on the same plans. A plan remembers its sorted edits and where the last rendered range started, so rendering a file range by range in ascending order (the bundler does, one statement at a time) is linear in the edits; ranges in any other order and edits added between two renderings give the same text, only slower.

2. The AST #

Nodes are structs (type start end props). Types are ESTree names in kebab-case keywords; properties are a plist. start/end delimit the source text of the node.

• Parentheses are kept as :parenthesized-expression nodes, so (a) satisfies T knows where (a) ends.
• Optional chains are marked with :optional t on each :member-expression and :call-expression link; there is no ESTree ChainExpression wrapper.
• Literals have their own types, as in Babel: :string-literal (:value), :numeric-literal / :bigint-literal (:raw; the internal numeric-value converts), :boolean-literal, :null-literal, :regexp-literal (:pattern :flags).
• Templates are :template-literal (:quasis :expressions). Each :template-element (:raw :cooked :tail) spans exactly its raw text. :cooked is NIL for an invalid escape, which is only allowed in tagged templates.
• Annotations (section 1): :pure t on an annotated :call-expression / :new-expression, :no-side-effects t on an annotated function; absent otherwise. analyze-scopes adds :scope, :binding and :reference properties holding its structs.

Main node types and their properties:

Example: defineProps<{ title: string; count?: number }>() is a :call-expression whose :type-arguments → :params → first element is a :ts-type-literal. Its :members are :ts-property-signature nodes with :key (:identifier "title"), :optional, and :type-annotation → :type-annotation (:ts-keyword-type "string").

3. Type stripping in detail #

The rules match Node 22's stripper (amaro 1.1, built on swc's fast TS strip). They are applied to our own AST and token stream:

• Blanking. Every erased character becomes whitespace of the same UTF-8 and UTF-16 length:

  • an ASCII character becomes a space;
  • a 2-byte character becomes U+00A0;
  • a 3-byte character becomes U+2002;
  • an astral character becomes a space followed by U+FEFF.

  Whitespace (tab, VT, FF, U+FEFF, and Space_Separator characters) and line terminators inside an erased range are kept. Comments inside an erased range are erased too.

• Erased statements. These statements are erased whole, including their ;:

  • type aliases and interfaces;
  • declare …, and module "x" / declare global blocks;
  • namespaces with only types;
  • overload signatures;
  • import type …, export type {…} and export type *.

  If the next token starts a new line with (, [, a template literal without substitutions, +, - or a regular expression, a ; is written in the statement's first column so that the following line does not join the previous one. When the previous token is already a ;, it is kept.

• as and satisfies. These erase from the end of the expression. If the next line starts with (, [ or a template literal, a ; is written over the as/satisfies keyword.

• Class members.

  • public, private, protected, readonly and override are erased unless the next token is on another line or cannot follow a modifier.
  • If such a modifier precedes a computed key, a *, or a key named in or instanceof, and the member is neither static nor decorated, a ; is left in its first column.
  • A typed field named get, set or static with no initializer gets a ; after its name.
  • Abstract members, declare fields, index signatures and method overloads are erased whole.

• Arrow functions.

  • When a return type starts on a later line than the ), the ) moves to the last column of the type, so that => stays on the parameters' line.
  • An async <T> (or return <T>, yield <T>, throw <T>) whose type parameters span lines turns the < into (.

• Bodies. If the body of an if, a loop or an else is an erased declaration, it becomes ;.

• enum. The declaration becomes var E = (function (E) { E[E["A"] = 0] = "A"; …; return E; })(E || {});.

  • Each member stays on its own line.
  • Initializers keep their source text.
  • References to members of the same enum, including members of earlier declarations of a merged enum, become E.member.
  • Auto-increment follows constant folding of integer initializers.
  • String-valued members (string literals, templates, string concatenation) get no reverse mapping.
  • declare enum is erased.

• Parameter properties. constructor(private x: T) gets a field declaration x; at the top of the class body and this.x = x;. The assignment goes after the super(…) statement in a derived class, otherwise at the top of the constructor body.

Deliberate differences from Node.

• Node rejects enum and parameter properties in strip mode; we transform them.
• Node passes <const>x and parameter decorators through, which yields JavaScript that does not parse. We report C123 and C124 instead.
• We erase export as namespace X (a UMD declaration in .d.ts files), which Node keeps.
• We accept the deprecated module Foo {} keyword (as an uninstantiated namespace), which Node 22.18+ rejects.

4. Scope rules (free identifiers) #

These rules serve rewrite-expression and free-identifiers; analyze-scopes (section 1) applies the same rules to whole modules and exposes the result.

• Scopes. Function scopes bind parameters, vars (hoisted, without entering nested functions), arguments, and a function expression's own name. Block scopes bind let/const/class/function/enum declarations and non-type imports. Catch clauses and for heads have their own scopes. A class expression binds its own name.
• Not references:
  • non-computed property keys, member names and method names;
  • labels;
  • import and export specifier names (except export { local });
  • TS type positions: annotations, type arguments and parameters, return types, implements, the types of as/satisfies, and declarations that are types only.
• Assignment targets are references (a = 1, [a, b] = c, ({ a } = c), a++). Binding patterns in declarations and parameters bind. Their default values and computed keys are references.

5. Errors #

js-syntax-error is a subtype of error with readers js-syntax-error-code, -message, -filename, -line (1-based), -column (1-based, in characters) and -offset. Its report reads:

src/App.ts:12:15: C101 Unexpected ';', expected expression
  const total = ;
                ^

6. Supported syntax and known gaps #

JavaScript (ES2024 and later).

• Declarations: let/const, using/await using, classes with private #x members (incl. #x in o), static blocks, accessors, accessor fields and decorators.
• Functions: async/await, generators, destructuring with defaults, spread/rest, labels, top-level await.
• Operators: optional chaining (a?.b, a?.[b], f?.()), ??, logical assignment (&&= ||= ??=), **.
• Literals: template literals (nested, tagged, raw/cooked), regular expression literals (the / ambiguity is resolved by the parser), numeric separators, BigInt, hex/octal/binary literals, Unicode escapes in identifiers and strings.
• Modules: import/export in all forms (incl. string names, export * as ns, with { type: "json" } and legacy assert), import() with options, import.meta, new.target, hashbang.
• Early errors that are enforced:
  • invalid assignment targets;
  • { a = 1 } outside patterns;
  • import/export outside module top level;
  • return outside functions;
  • await/yield as identifiers in modules;
  • with and legacy octal in strict code;
  • uninitialized const;
  • a tagged template in an optional chain.

TypeScript 5.x.

• Type annotations and the full type grammar:
  • unions and intersections (with a leading | or &);
  • conditional types with infer … extends;
  • mapped types with +/-/as;
  • template literal types, tuples (named, optional, rest), function and constructor types (abstract new);
  • typeof queries (with type arguments), import() types (with attributes), keyof/unique symbol/readonly, indexed access;
  • type predicates (x is T, asserts x is T, this is T).
• Interfaces, type aliases, generics with constraints, defaults and const/in/out, and type parameters on arrow functions.
• Expressions: call and new type arguments, instantiation expressions, as, as const, satisfies and !.
• Declarations: declare, abstract, accessibility modifiers, readonly, override, ? and !: on members, this parameters, overloads, decorators, accessor, enum/const enum, namespace/module/ declare global/declare module "x", parameter properties.
• Module syntax: import type, export type, inline type specifiers (with all the type as as cases), import x = require(), export = and export as namespace.
• Ambiguities are decided the way tsc decides them. For example, a < b > (c) is a call in TypeScript and comparisons in JavaScript, and in a ? (b) : c => d the arrow has no return type.

Known gaps (none of these is needed by the framework):

• No JSX/TSX.
• The bodies of regular expression literals are not validated (only their flags are).
• Some early errors are not reported:
  • duplicate parameters or declarations;
  • break/continue label targets;
  • new.target and super placement;
  • strict-mode reserved words used as binding names;
  • getter and setter arity.
• CommonJS-only top-level return is a syntax error. Node accepts it in .cjs/CJS files.
• import defer / import source (stage 3) are parsed only in their simplest form.
• Computed import() arguments are not reported by module-imports.
• A namespace with runtime code is rejected (C120) rather than transformed.

7. Testing, coverage and performance #

bin/test-system roots-parse/tests runs 79 tests. They cover:

• unit tests per construct (parse-lexer-*, parse-js-*, parse-ts-*), annotation comments included;
• 111 strip cases whose expected output was generated by Node (parse-strip-matches-node);
• transform tests that execute the output in Node;
• scope and analysis tests: the template analyses, fifteen parse-scope-* tests of the scope tree, the CommonJS test and edit plans;
• the corpus and performance tests: differential strip and span tests, the scope tree against free-identifiers over the sampled modules, the 36 PURE and 15 NO_SIDE_EFFECTS annotations of @vue/reactivity, and CommonJS detection on semver, ajv and acorn.

The corpus tests are differential:

• TypeScript. Every .ts/.mts/.cts file of the repository outside node_modules, plus up to 500 non-declaration TypeScript files sampled evenly from node_modules (the repository pins zod and ajv as development dependencies because they ship their TypeScript sources), go through both strip-types and Node's stripTypeScriptTypes (via tests/oracle.cjs). Files Node can strip must be byte-identical. Files Node refuses must be transformed into JavaScript that parses, or be refused with a C12x code. Files Node finds invalid must fail for us too.
• JavaScript. 150 .js/.mjs/.cjs files sampled from node_modules must parse. The spans of every identifier, literal and template element must equal those of acorn's ESTree.

ROOTS_PARSE_CORPUS=full uses all of node_modules, .d.ts files included.

Results on this repository (Node 22.23, amaro 1.1.8):

Until 0.2.0 the corpus also held the TypeScript sources of the vuecli repository this project started in (2319/2319 byte-identical in the full run). | full: all node_modules JS | 2667 | 2667/2667 parse; 2667/2667 identical spans to acorn |

Performance (SBCL 2.2.9, one core):

parse-performance fails when parsing takes more than a second per megabyte.

8. Design notes #

• Scanner driven by the parser (lexer.lisp), as in tsc. It only produces /, > and }. The parser re-scans where the grammar needs more:
  • rescan-slash produces a regular expression where an expression starts;
  • rescan-greater produces >= and >> in expressions only, so Array<Array<T>> closes two type argument lists;
  • rescan-template-continuation continues a template after }.
• Speculation. lookahead always rewinds. try-parse keeps what its body parsed when the body succeeds; a syntax error inside it aborts the attempt (catch/throw, no condition objects). Arrow functions and type arguments in expressions follow tsc's decision procedures (isParenthesizedArrowFunctionExpression, canFollowTypeArgumentsInExpression), including tsc's memo of positions where an arrow failed.
• Token record. strip-types keeps every token (kind, span, preceding line break). The ASI and class-modifier rules look at neighbouring tokens, exactly like swc does.
• Edit plans (strip.lisp) collect four kinds of edits: a bit-vector of erased characters, single-character overwrites, insertions, and range replacements. Rendering them is one pass over the source: replacements are sorted by start (the larger, then the later, first), a replacement starting inside the one just written is nested and dropped unless it reaches past it, which is the overlap error. rewrite-expression adds its _ctx. insertions to the same plan as the type erasure. The whitespace-preserving blanking and the overwrites are the stripper's own; the public API (insert, replace, erase, render) is what the bundler edits module text with.
• Annotation comments. skip-trivia tests every comment it skips: after an optional ! and whitespace the text must start with @/# and one of the two markers, at a word boundary. A hit is kept in the parser's ann slot (part of the speculation state) for the token that follows; the parser reads it where the grammar knows the node kind — parse-lhs marks the outermost call/new of the chain it just parsed, parse-primary and parse-assignment mark function expressions and arrows, and a statement's annotation is pushed down through export/export default/const f = … by mark-no-side-effects. The token record is untouched and the cost per token is one slot write, so parsing speed is unchanged.
• Scope trees. One walk opens scopes and declares bindings while queueing every identifier use; the queue is sorted by position and resolved after the walk, when all hoisted declarations exist (this is also where the implicit arguments is created). Scope lookup tables start as alists and become hash tables past eight names. The walk is a second implementation of the rules behind free-identifiers and is checked against it on the corpus; it processes vite's 1.4 MB node.js in about 0.1 s.
• Robustness.
  • Long left-associative operator chains (a + b + …, 50 000 terms in the tests) are walked iteratively.
  • Nesting deeper than 1000 levels is a C109 error.
  • Exhausting the stack in any entry point also becomes C109, analyze-scopes included: member and call chains are parsed in a loop and do not count as nesting, but the scope walk recurses along them.
  • The parser never evaluates input.