Security hardening: what was fixed

Result #

• Every critical, high and medium finding is fixed by default, except PM-06, which is closed only by two opt-in switches (reason below).
• Full suite in the worktree: 472 tests, 471 passed; the one failure is e2e-extension-popup (known, environmental). Log: sec/full-suite.log.
• The full-suite script only loads systems named …/tests, so two systems were run separately afterwards:
  • roots-pm-tests: 64 of 64 passed (one run; see "SBCL abort" under Open).
  • roots-data-tests: 38 of 39 passed. The failure is data-e2e-notes-extension, the same ERR_BLOCKED_BY_CLIENT as e2e-extension-popup.
• The nine browser tests of roots/data that were PENDING for every owner now ran: I put the real @sqlite.org/sqlite-wasm 3.54.1-build1 (hashes equal to the pins) into examples/notes/node_modules/ of the worktree (git-ignored; still there). They found one real bug in the WASM worker, fixed below.
• All examples build for all six targets (todo, notes, gallery, hello-lisp × web, pwa, ext, ios, android, term: 24 builds, exit 0) into integ/ex/. browser-counter and hello.lisp have no roots.lisp and were not built.
• Android Kotlin (shell core, SqlitePlugin.kt, StoragePlugin.kt, the four test classes) compiles against android-36 with Android Studio's kotlinc; JUnit: 34 tests OK. This covers the data owner's and re-tester's Kotlin, which had never been compiled.
• Not verified anywhere: behaviour on an Android device or in a WKWebView (gesture rule, frame reply, backup flags), Gradle and Xcode builds, Linux, browsers other than Chromium.

What the integrator changed #

Applied from "needsFromOthers" and "stillOpen":

Hot paths: nothing was added to runtime/core, runtime/dom or runtime/reactivity by the integration. The worker change is one try per statement run; the roots-data policy is created once per page.

Decisions taken as secure defaults #

These change behaviour for existing apps. Each has a documented way back.

• Trusted Types on for web, pwa and the Roots shells. An app that ships a library which assigns innerHTML sets :csp (:trusted-types nil).
• Native backups off. :native (:backup t) turns them on.
• Native plugin allow-list derived from imports. An app that calls RootsNative.call(…) itself lists :native (:plugins (…)); otherwise it gets BRIDGE_UNKNOWN_PLUGIN.
• A package's Lisp is refused (B129), no opt-in.
• Another SQLite WASM version is an error (D164) unless pinned in roots.lisp.

Decisions for you #

• PM-06 defaults. Whether :builds (approvals need a line in roots.lisp) and :verify-age should be on by default. Left opt-in: the first breaks CI for every project with approved install scripts until it adds the key, the second adds a registry request per package to every install.
• import {} from "pkg" now counts as a side-effect import and keeps the module (the re-tester reversed the owner's test).
• R303 / R304 / R306 throw into the error handler instead of warn-and-drop.
• iOS file protection for the database is left at the platform default (see NE-3).

Findings #

Status: fixed, mitigated (closed in part or by an option), documented, open.

Build, bundler, parser #

Dev server #

Package manager #

Templates and runtime #

Targets and native shells #

Data #

Open #

• PM-06 (medium): opt-in only, see above.
• SBCL abort in the pm suite. The re-tester saw SIGABRT (pointer being freed was not allocated, in the cleanup of sb-posix:stat/lstat) in 4 of 15 runs under load; the pm owner's GC call in one test does not cover it. Root cause unknown; whether a real roots install can hit it is unproven. My one run passed. Next step: a newer SBCL, or a breakpoint on malloc_error_break.
• Device checks for the Android and iOS shell changes (NE-3, NE-8, NE-11, NE-15) and the Android sqlite rules (NE-2).
• New, from the data re-test, documented only: a tx statement written INSERT OR ROLLBACK (or RAISE(ROLLBACK)) ends the transaction; a callback that catches the error and goes on writes in autocommit.
• Hardening left as it is: BB-07 annotation warning, BB-09 slow passes, BB-11 authenticated cache entries, PM-07 index authentication, PM-13 sha1 upgrade, DATA-12 row limit, DATA-13 defensive mode on other drivers, Trusted Types for the extension and Capacitor, NE-10 shipped Gradle metadata, NE-18.
• Small: write-json in src/build/util.lisp prints doubles with ~f (the data command no longer uses it); docs/runtime.md says R304 covers every name starting with on, the code exempts the bare name on; a static on-click= attribute compiles and throws R304 at run time.

Behaviour changes for app authors #

Build and config

• New roots.lisp keys: :csp, :pwa (:precache-exclude …), :native (:backup …), :data (:sqlite-version … :sqlite-mjs-sha256 … :sqlite-wasm-sha256 …), :packages (:builds … :verify-age …).
• Pages of web, pwa and the Roots shells require Trusted Types (:csp (:trusted-types nil) turns it off). Extra origins for fetch, images, fonts, media and frames go in :csp.
• A dependency that is Lisp is B129. Importing into node_modules by path, or a node_modules/<name> linking outside the project (npm link), is B125. A package without a valid package.json name and version is B128.
• A name declared twice in a module is C110, in bundled and unbundled builds. <!-- in module code is C111.
• import "pkg" and import {} from "pkg" in app code keep the module despite "sideEffects": false.
• Minified output has other short names where a var sits in a block or catch clause. catch (e) { var e } keeps its name; a clash is B137.
• Symbolic links: inside the output directory B141; public/, icon.png, migrations/, native/<platform>/ as links B163. More key and secret files are skipped from public/ (B164).
• roots/data on DOM targets: another @sqlite.org/sqlite-wasm version is an error (D164) unless pinned; changed files are D166.
• Errors are one coded line (B191, B192, src/x.js:1:23: C102 …); ROOTS_DEBUG=1 shows the backtrace. ROOTS_NO_CACHE=1 turns the compile cache off; a cache directory others can write is not used (B167).
• bin/roots and bin/test-system load only roots.asd, src/*/*.asd, runtime/*/*.asd.

Dev server

• Host needs the port. Other sites cannot frame or embed dev-server files. The reload channel is for the page only, at most 32 at once. Time limits on request heads (5 s) and stalled responses. The --host token cookie is roots_token_<port>. Internal errors during a rebuild show in the overlay and roots dev keeps running.

Package manager

• A symlinked node_modules (or .roots, .bin, a package directory) is PM152. Invalid dependency names are PM138 / PM172 / PM140. Names differing only in case are PM153.
• Project .npmrc: ${VAR} in a registry URL is PM170, in a credential it warns (PM171); credential keys match host, port and path segments exactly. Authorization is dropped on a redirect to another port or scheme. An https:// proxy is refused (PM107).
• Unpacked size per package is 512 MB (was 2 GB). Stricter roots.lock checks (PM142, PM146, PM137). Plain http only to localhost and 127.x.x.x.
• macOS: OpenSSL only from Homebrew, MacPorts or an absolute $ROOTS_OPENSSL_DIR. A library path variable with an empty or relative element is refused (PM101; D170 for roots db). On Linux a leading or trailing colon in LD_LIBRARY_PATH is such an element.
• New: roots install --verify, --verify-age; roots doctor prints the library files.

Templates and runtime

• Attribute names are judged in any case, also when they come from data: onClick strings throw R304, srcDoc / InnerHTML throw R303, javascript: under HREF is dropped (R307). data: is refused as the document of iframe, frame, embed and object.
• <component :is="string"> refuses script, style, iframe, frame, frameset, object, embed, base, meta and link (R306) unless a component of that name is registered.
• :srcDoc, :src-doc, :InnerHTML are C211; static URLs with a tab or newline in the scheme and <object data="javascript:…"> are C224; <style> under v-pre is C208.
• v-bind="obj" binds own keys only. Invalid attribute names are dropped with R308.
• route.params and route.query have no prototype. Router targets never leave the app; a refused pushState resolves false (R407).

PWA and native

• The service worker install fails if a precached file is missing or not byte-identical to the build (a host that rewrites files stops updates); updates.onError(message) reports it. Cache names changed, so users download the precache once more. Host files such as _headers and CNAME are not precached.
• Native: plugins derived from imports, backups off, a bridge of another ABI version or a frame gets BRIDGE_UNAVAILABLE, script cannot open the browser without a user gesture (use openUrl), iOS link previews off, the database file is <name>.sqlite3 (an old <name> is renamed on first open). Extension builds warn about broad permissions (B158).

roots/data

• D113 (value count), D114 (second statement), D112 (NaN, Infinity), D111 (NUL in SQL), D117 (ATTACH, DETACH, VACUUM INTO without { attach: true }), D115 (undefined in where), D136 (transaction control in a migration), D154 ({ memoryFallback: false }).
• trusted_schema is off ({ trustedSchema: true } turns it on). Integers beyond 2^53 come back as decimal strings. A __proto__ column is an ordinary property. Live queries also fire for failed writes and trigger writes. On Bun, scripts stop at the first failing statement.
• roots db query escapes control characters; existing database files are tightened to 0600; migrate and types cannot ATTACH.

New diagnostic codes #

Environment variables: ROOTS_NO_CACHE, ROOTS_DEBUG, ROOTS_OPENSSL_DIR.

Draft CHANGELOG section #

## security hardening

A security audit of the whole tree (dev server, package manager, templates and runtime,
native shells and targets, build and bundler, data), every finding checked by a second
reviewer, then fixed and re-tested. After verification: one critical, four high and eight
medium findings; all are fixed by default except one (roots.lock approvals, below). Each fix has a regression
test. Several defaults change; each has a documented way back.

- **A package's Lisp no longer runs in the toolchain.** A dependency that resolved to a
  `.lisp` file, or a `.vue` with `<script lang="lisp">`, was compiled like app code, and
  compiling Lisp runs its macros: any package could run Common Lisp as the developer on
  every `roots build` and `roots dev`. It is now B129, with no opt-in; while a package file
  is compiled the compiler also refuses `(lisp …)`, `defmacro` and `macrolet`. Importing
  into `node_modules` by path, and a package directory that links outside the project, are
  B125. docs/build.md, "What a build trusts", says what a build trusts (your sources:
  completely; packages: not at all).
- **Minifier: a `var` in a block or catch clause could get the short name of a binding
  around it**, which changed what the program did (the audit's example made an
  `isAllowed` check pass for a forged token). The scope model now records where hoisted variables are
  written; the bundler's own renaming had the same hole. A catch clause with a `let` /
  `const` in its body and an unused parameter minified to code that did not load. The
  minifier's output is now compared with its input scope by scope before it is written.
  Short names differ from 0.3.0 for such code; `catch (e) { var e }` keeps its name (B137
  when it cannot).
- **Output directory:** `roots build` wrote and deleted through symbolic links inside it,
  so a checkout could carry a link that made a build overwrite files elsewhere. Every write
  and delete now walks the path without following links (B141), also for the dev server's
  error page. `public/`, `icon.png`, `migrations/` and `native/<platform>/` are never read
  through a link (B163), and more key material is kept out of `public/` (B164).
- **Package manager:** OpenSSL, zlib and SQLite are loaded by absolute path on macOS (a
  library in the working directory used to be picked up by `roots why`); a symlinked
  `node_modules/.roots` no longer makes `roots install` delete the link's target (PM152);
  dependency names are validated everywhere (`"../../src": "npm:x"` deleted directories;
  PM138); a git, URL or file entry of roots.lock must resolve where package.json says
  (PM142). Also: no heap exhaustion from a 25-byte gzip, Authorization stays on its origin,
  the request line cannot be extended, project `.npmrc` files are not trusted with the
  environment, loopback is an address, names differing in case are refused (PM153).
  **Not closed by default:** a change to roots.lock alone can still approve an install
  script or pin a version inside the cooldown; `:packages (:builds (…))` and `:verify-age t`
  (or `roots install --verify-age`) close it for projects that set them.
- **SQLite WASM is pinned by hash.** roots/data shipped `index.mjs` and `sqlite3.wasm` from
  whatever `node_modules` was found above the project. The build now ships them only when
  their SHA-256 is the pinned one (D166); another version is an error (D164, was a warning)
  unless roots.lisp pins it under the new `:data` key.
- **PWA updates are checked.** The service worker fetched its files through the HTTP cache
  and never compared them with the build, so an update could be assembled from stale files.
  Every file is now fetched past the cache with its SHA-256; a failed install keeps the
  running version and reports through the new `updates.onError`. Two apps on one origin no
  longer delete each other's caches.
- **Templates and the DOM backend:** the attribute rules were case-sensitive, so `onClick`,
  `srcDoc` or `HREF` from data (a spread, fallthrough attributes, `<component :is>` props)
  got past them. Names are now judged in lower case, once per name (a Map lookup per write
  in place of a chain of string comparisons). `<component :is>` refuses strings naming
  document-level tags (R306), the router never renders a link out of the app, route params
  and query have no prototype, and invalid attribute names are dropped (R308).
- **Trusted Types.** Pages of web, pwa and the Roots shells are written with
  `require-trusted-types-for 'script'`; the only policies are the runtime's own for the
  service worker and the roots/data worker URL. `:csp` in roots.lisp adds origins to the
  policy (never to `script-src`) and `:csp (:trusted-types nil)` turns the requirement off.
- **Dev server:** bounded request heads, responses and connection slots; at most 32 reload
  channels, for the page only; no socket is written while the server lock is held; no
  framing by other origins; `Host` needs the port; internal errors during a rebuild no
  longer end `roots dev`.
- **roots/data:** `roots db types` and `migrate --check` could write arbitrary files through
  a migration (ATTACH); toolchain connections cannot attach now, and a migration with
  `BEGIN` / `COMMIT` / `ROLLBACK` is refused before anything runs (D136). At run time ATTACH
  needs `{ attach: true }` (D117), `trusted_schema` is off, a wrong parameter count (D113),
  a second statement (D114), NaN and NUL are refused on every driver, integers beyond 2^53
  come back as decimal strings, live queries see writes made by triggers and by failing
  statements, and `roots db query` cannot repaint the terminal. The WASM worker ran a
  statement's stale error instead of the statement after a failure; fixed.
- **Native shells:** the plugin allow-list is what the app imports (it was all five
  built-ins), app data is excluded from backups unless `:native (:backup t)`, the sqlite
  plugin refuses ATTACH and VACUUM INTO, the bridge checks its ABI version and answers
  frames with `BRIDGE_UNAVAILABLE`, script cannot leave the app without a user gesture, logs
  name scheme and host only. Android and iOS code was compiled and unit-tested; nothing ran
  on a device.
- **Parser and build errors:** a name declared twice is C110, `<!--` in a module C111, deep
  nesting C109 or B192, a `.js` syntax error one line with file and position, anything
  unexpected B191 (`ROOTS_DEBUG=1` for the backtrace). `import "pkg"` keeps the module in a
  bundle whatever the package's `sideEffects` says.
- **New codes:** B129, B137, B158, B167, B191, B192; C110, C111; R308, R407; PM138, PM153,
  PM163; D113, D114, D117, D136, D154, D166. **New roots.lisp keys:** `:csp`, `:data`,
  `:pwa (:precache-exclude)`, `:native (:backup)`, `:packages (:builds :verify-age)`.
  **Environment:** `ROOTS_NO_CACHE`, `ROOTS_DEBUG`, `ROOTS_OPENSSL_DIR`.
- **Tests:** 472 in the `…/tests` systems (471 pass here; `e2e-extension-popup` cannot load
  an extension on this machine), 64 in the package manager's, 39 in roots/data's (the
  extension e2e fails the same way). roots-build's tests now load roots-pm.
- **Open:** the roots.lock default above; an intermittent SBCL abort in the package
  manager's test suite under load, cause unknown; device runs of the shell changes; no
  row limit in roots/data; cache entries and the store index are not authenticated.