Skip to content

Commit

Permalink
Update tree-sitter (etc)
Browse files Browse the repository at this point in the history
This change updates tree-sitter and related dependencies, and accounts for the current iteration of churn in various special case details of how a tree-sitter project is supposed to work. On the bright side, it turns out things got a lot simpler this time!

- The structure of the `@getodk/tree-sitter-xpath` package is now much more like the other packages. This is largely because the assumptions made by `tree-sitter-cli` are based around a special `tree-sitter.json` file, rather than overloading `package.json` with tree-sitter specific details.

    TODO: it’s possible we can now switch the sub-package to ESM (as all of the other sub-packages are). I would have already looked into this, but there was more than enough churn to get back to a working state.

- We no longer (need to) build for Node. Since all of our downstream packages consume the parser as WASM, we benefit from a simpler and (MUCH) faster build step. Note that this may technically be a breaking change (though I’m not sure that it is, since I’m pretty sure we never published the Node bindings, even when they were built).

- Many of the extraneous build artifacts are no longer produced, either because they’re excluded in `tree-sitter.json` or because we are no longer building for Node at all. This allows us to significantly scale down custom build scripting, as well as tree-sitter specific entries in .gitignore/.gitattributes.

- Note: `tree-sitter.json` and the `test` directory have been moved into the Vite-default `public` directory, so they automatically get copied into the `dist` build directory. Then all `tree-sitter` commands are run from that directory. This maximizes our ability to use existing tooling and conventions for the sub-package, and ensures that all ephemeral build product is contained in one (conventional) place that gets cleaned up like any other project on rebuild.
  • Loading branch information
eyelidlessness committed Dec 17, 2024
1 parent 7ea1dc7 commit c5d5982
Show file tree
Hide file tree
Showing 22 changed files with 169 additions and 353 deletions.
10 changes: 0 additions & 10 deletions .gitattributes
Original file line number Diff line number Diff line change
Expand Up @@ -3,13 +3,3 @@
*.jpg -text

*.xlsx binary

packages/tree-sitter-xpath/src/*.json linguist-generated
packages/tree-sitter-xpath/src/parser.c linguist-generated
packages/tree-sitter-xpath/src/tree_sitter/* linguist-generated

packages/tree-sitter-xpath/bindings/** linguist-generated
packages/tree-sitter-xpath/binding.gyp linguist-generated
packages/tree-sitter-xpath/setup.py linguist-generated
packages/tree-sitter-xpath/Makefile linguist-generated
packages/tree-sitter-xpath/Package.swift linguist-generated
72 changes: 24 additions & 48 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -58,12 +58,9 @@ jobs:
examples/*/dist
packages/*/dist
packages/web-forms/dist-demo
packages/tree-sitter-xpath/grammar.js
packages/tree-sitter-xpath/src/grammar.json
packages/tree-sitter-xpath/src/parser.c
packages/tree-sitter-xpath/src/tree_sitter/parser.h
packages/tree-sitter-xpath/tree-sitter-xpath.wasm
packages/tree-sitter-xpath/types
packages/tree-sitter-xpath/dist/grammar.js
packages/tree-sitter-xpath/dist/tree-sitter.json
packages/tree-sitter-xpath/dist/tree-sitter-xpath.wasm
key: build-${{ matrix.node-version }}-${{ github.sha }}

# `@getodk/xpath` tests (currently) expect this time zone
Expand Down Expand Up @@ -109,12 +106,9 @@ jobs:
examples/*/dist
packages/*/dist
packages/web-forms/dist-demo
packages/tree-sitter-xpath/grammar.js
packages/tree-sitter-xpath/src/grammar.json
packages/tree-sitter-xpath/src/parser.c
packages/tree-sitter-xpath/src/tree_sitter/parser.h
packages/tree-sitter-xpath/tree-sitter-xpath.wasm
packages/tree-sitter-xpath/types
packages/tree-sitter-xpath/dist/grammar.js
packages/tree-sitter-xpath/dist/tree-sitter.json
packages/tree-sitter-xpath/dist/tree-sitter-xpath.wasm
key: build-${{ matrix.node-version }}-${{ github.sha }}
fail-on-cache-miss: true

Expand Down Expand Up @@ -166,12 +160,9 @@ jobs:
examples/*/dist
packages/*/dist
packages/web-forms/dist-demo
packages/tree-sitter-xpath/grammar.js
packages/tree-sitter-xpath/src/grammar.json
packages/tree-sitter-xpath/src/parser.c
packages/tree-sitter-xpath/src/tree_sitter/parser.h
packages/tree-sitter-xpath/tree-sitter-xpath.wasm
packages/tree-sitter-xpath/types
packages/tree-sitter-xpath/dist/grammar.js
packages/tree-sitter-xpath/dist/tree-sitter.json
packages/tree-sitter-xpath/dist/tree-sitter-xpath.wasm
key: build-${{ matrix.node-version }}-${{ github.sha }}
fail-on-cache-miss: true

Expand Down Expand Up @@ -231,12 +222,9 @@ jobs:
examples/*/dist
packages/*/dist
packages/web-forms/dist-demo
packages/tree-sitter-xpath/grammar.js
packages/tree-sitter-xpath/src/grammar.json
packages/tree-sitter-xpath/src/parser.c
packages/tree-sitter-xpath/src/tree_sitter/parser.h
packages/tree-sitter-xpath/tree-sitter-xpath.wasm
packages/tree-sitter-xpath/types
packages/tree-sitter-xpath/dist/grammar.js
packages/tree-sitter-xpath/dist/tree-sitter.json
packages/tree-sitter-xpath/dist/tree-sitter-xpath.wasm
key: build-${{ matrix.node-version }}-${{ github.sha }}
fail-on-cache-miss: true

Expand Down Expand Up @@ -296,12 +284,9 @@ jobs:
examples/*/dist
packages/*/dist
packages/web-forms/dist-demo
packages/tree-sitter-xpath/grammar.js
packages/tree-sitter-xpath/src/grammar.json
packages/tree-sitter-xpath/src/parser.c
packages/tree-sitter-xpath/src/tree_sitter/parser.h
packages/tree-sitter-xpath/tree-sitter-xpath.wasm
packages/tree-sitter-xpath/types
packages/tree-sitter-xpath/dist/grammar.js
packages/tree-sitter-xpath/dist/tree-sitter.json
packages/tree-sitter-xpath/dist/tree-sitter-xpath.wasm
key: build-${{ matrix.node-version }}-${{ github.sha }}
fail-on-cache-miss: true

Expand Down Expand Up @@ -361,12 +346,9 @@ jobs:
examples/*/dist
packages/*/dist
packages/web-forms/dist-demo
packages/tree-sitter-xpath/grammar.js
packages/tree-sitter-xpath/src/grammar.json
packages/tree-sitter-xpath/src/parser.c
packages/tree-sitter-xpath/src/tree_sitter/parser.h
packages/tree-sitter-xpath/tree-sitter-xpath.wasm
packages/tree-sitter-xpath/types
packages/tree-sitter-xpath/dist/grammar.js
packages/tree-sitter-xpath/dist/tree-sitter.json
packages/tree-sitter-xpath/dist/tree-sitter-xpath.wasm
key: build-${{ matrix.node-version }}-${{ github.sha }}
fail-on-cache-miss: true

Expand Down Expand Up @@ -421,12 +403,9 @@ jobs:
examples/*/dist
packages/*/dist
packages/web-forms/dist-demo
packages/tree-sitter-xpath/grammar.js
packages/tree-sitter-xpath/src/grammar.json
packages/tree-sitter-xpath/src/parser.c
packages/tree-sitter-xpath/src/tree_sitter/parser.h
packages/tree-sitter-xpath/tree-sitter-xpath.wasm
packages/tree-sitter-xpath/types
packages/tree-sitter-xpath/dist/grammar.js
packages/tree-sitter-xpath/dist/tree-sitter.json
packages/tree-sitter-xpath/dist/tree-sitter-xpath.wasm
key: build-${{ matrix.node-version }}-${{ github.sha }}
fail-on-cache-miss: true

Expand Down Expand Up @@ -476,12 +455,9 @@ jobs:
examples/*/dist
packages/*/dist
packages/web-forms/dist-demo
packages/tree-sitter-xpath/grammar.js
packages/tree-sitter-xpath/src/grammar.json
packages/tree-sitter-xpath/src/parser.c
packages/tree-sitter-xpath/src/tree_sitter/parser.h
packages/tree-sitter-xpath/tree-sitter-xpath.wasm
packages/tree-sitter-xpath/types
packages/tree-sitter-xpath/dist/grammar.js
packages/tree-sitter-xpath/dist/tree-sitter.json
packages/tree-sitter-xpath/dist/tree-sitter-xpath.wasm
key: build-${{ matrix.node-version }}-${{ github.sha }}
fail-on-cache-miss: true

Expand Down
98 changes: 0 additions & 98 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -25,104 +25,6 @@ packages/**/dist
# Generated documentation
packages/*/api-docs

## ============================================================================
## Specific to tree-sitter and/or @getodk/tree-sitter-xpath
##
## The tree-sitter-xpath build process generates a slew of files, many of
## which we ignore. This section includes two sub-sections:
##
## 1. A set of ignores derived from tree-sitter's own generated .gitignore
## file, with its patterns adjusted to their place in the
## tree-sitter-xpath package directory.
##
## 2. A set of additive ignore patterns which are not addressed by those
## derived patterns, but which we also regard as either build artifacts
## or detritus left behind in their creation.
##
## Not specified here: a few **other files** are generated by tree-sitter
## which are not appropriate to ignore, because their presence would cause
## conflicts with editor and/or version control. For more detail, see:
##
## ./packages/tree-sitter-xpath/scripts/build/shared.mjs
## ============================================================================

# ----------------------------------------------------------------------------
# 1. Derived from the .gitignore generated by tree-sitter
# ----------------------------------------------------------------------------

# Rust artifacts
packages/tree-sitter-xpath/Cargo.lock
packages/tree-sitter-xpath/target/

# Node artifacts
packages/tree-sitter-xpath/build/
packages/tree-sitter-xpath/prebuilds/
packages/tree-sitter-xpath/node_modules/
packages/tree-sitter-xpath/*.tgz

# Swift artifacts
packages/tree-sitter-xpath/.build/

# Go artifacts
packages/tree-sitter-xpath/go.mod
packages/tree-sitter-xpath/go.sum
packages/tree-sitter-xpath/_obj/

# Python artifacts
packages/tree-sitter-xpath/.venv/
packages/tree-sitter-xpath/dist/
packages/tree-sitter-xpath/*.egg-info
packages/tree-sitter-xpath/*.whl

# C artifacts
packages/tree-sitter-xpath/*.a
packages/tree-sitter-xpath/*.so
packages/tree-sitter-xpath/*.so.*
packages/tree-sitter-xpath/*.dylib
packages/tree-sitter-xpath/*.dll
packages/tree-sitter-xpath/*.pc

# Example dirs
packages/tree-sitter-xpath/examples/*/

# Grammar volatiles
packages/tree-sitter-xpath/*.wasm
packages/tree-sitter-xpath/*.obj
packages/tree-sitter-xpath/*.o

# ----------------------------------------------------------------------------
# 2. Additional ignore patterns beyond those generated by tree-sitter
#
# Note that some of these may be redundant to those above. Most of these
# patterns predate tree-sitter's generation of .gitignore. Keeping them intact
# may make it easier to untangle this nonsense if tree-sitter stops generating
# such files on every build.
# ----------------------------------------------------------------------------

packages/tree-sitter-xpath/bindings/
packages/tree-sitter-xpath/build/
packages/tree-sitter-xpath/src/tree_sitter
packages/tree-sitter-xpath/src/grammar.json
packages/tree-sitter-xpath/src/node-types.json
packages/tree-sitter-xpath/src/parser.c
packages/tree-sitter-xpath/target/
packages/tree-sitter-xpath/types/tree-sitter-xpath-parser.d.ts
packages/tree-sitter-xpath/binding.gyp
packages/tree-sitter-xpath/Cargo.lock
packages/tree-sitter-xpath/Cargo.toml
packages/tree-sitter-xpath/grammar.js
packages/tree-sitter-xpath/grammar.js.map
packages/tree-sitter-xpath/Makefile
packages/tree-sitter-xpath/Package.swift
packages/tree-sitter-xpath/pyproject.toml
packages/tree-sitter-xpath/setup.py
packages/tree-sitter-xpath/tree-sitter-xpath.wasm


## ============================================================================
## End of patterns specific to tree-sitter and/or @getodk/tree-sitter-xpath
## ============================================================================

# Test artifcats
## Playwright, apparently
packages/*/playwright-report
Expand Down
1 change: 1 addition & 0 deletions eslint.config.js
Original file line number Diff line number Diff line change
Expand Up @@ -407,6 +407,7 @@ export default tseslint.config(
'packages/*/playwright.config.ts',
'packages/*/vite.config.ts',
'packages/*/vitest.config.ts',
'packages/tree-sitter-xpath/vite.config.mts',
'packages/xforms-engine/vite.*.config.ts',
'packages/*/tools/**/*',
'packages/tree-sitter-xpath/scripts/build/*.mjs',
Expand Down
2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,6 @@
"vue-tsc": "^2.0.19"
},
"resolutions": {
"**/tree-sitter": "0.21.1"
"**/tree-sitter": "0.22.1"
}
}
2 changes: 1 addition & 1 deletion packages/tree-sitter-xpath/index.d.ts
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
/// <reference path="./types/tree-sitter-xpath-parser.d.ts" />
/// <reference path="./dist/tree-sitter-xpath-parser.d.ts" />

export * from '@getodk/tree-sitter-xpath/parser';
40 changes: 15 additions & 25 deletions packages/tree-sitter-xpath/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -14,48 +14,38 @@
"homepage": "https://getodk.org/",
"files": [
"README.md",
"grammar.js",
"index.d.ts",
"types/tree-sitter-xpath-parser.d.ts",
"tree-sitter-xpath.wasm"
"dist/grammar.js",
"dist/tree-sitter-xpath-parser.d.ts",
"dist/tree-sitter-xpath.wasm",
"index.d.ts"
],
"engines": {
"node": "^18.20.5 || ^20.18.1 || ^22.12.0",
"yarn": "1.22.22"
},
"scripts": {
"build": "yarn build:pre && yarn build:js && yarn build:node && yarn build:parser-types && yarn build:wasm && yarn build:post",
"build": "npm-run-all -nl build:*",
"build:pre": "node ./scripts/build/pre.mjs",
"build:node": "tree-sitter generate && node-gyp rebuild && node ./scripts/build/post-node.mjs",
"build:post": "node ./scripts/build/post.mjs",
"build:js": "vite build",
"build:tree-sitter": "cd dist && tree-sitter generate && tree-sitter build --wasm",
"build:parser-types": "node ./scripts/build/parser-types.mjs",
"build:wasm": "tree-sitter build --wasm",
"install": "true # Prevent invoking `node-gyp rebuild` on `yarn install`",
"test": "npm-run-all -nl test:*",
"test:types": "tsc --project ./tsconfig.json",
"test:grammar": "tree-sitter test"
"test:grammar": "cd dist && tree-sitter test --wasm"
},
"devDependencies": {
"@asgerf/dts-tree-sitter": "^0.1.0",
"nan": "2.20.0",
"node-gyp": "10.2.0",
"tree-sitter": "0.21.1",
"tree-sitter-cli": "0.23.0",
"nan": "2.22.0",
"tree-sitter": "0.22.1",
"tree-sitter-cli": "0.24.5",
"vite": "^5.4.8",
"web-tree-sitter": "0.23.0"
"web-tree-sitter": "0.24.5"
},
"peerDependencies": {
"web-tree-sitter": "0.23.0"
"web-tree-sitter": "0.24.5"
},
"resolutions": {
"**/tree-sitter": "0.21.1",
"@asgerf/dts-tree-sitter/tree-sitter": "0.21.1"
},
"tree-sitter": [
{
"scope": "source.xpath",
"injection-regex": "^xpath$"
}
]
"**/tree-sitter": "0.22.1",
"@asgerf/dts-tree-sitter/tree-sitter": "0.22.1"
}
}
34 changes: 34 additions & 0 deletions packages/tree-sitter-xpath/public/tree-sitter.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
{
"$schema": "https://tree-sitter.github.io/tree-sitter/assets/schemas/config.schema.json",
"grammars": [
{
"name": "@getodk/xpath",
"camelcase": "GetODKXPath",
"scope": "source.xpath",
"path": ".",
"file-types": null,
"injection-regex": "^xpath$"
}
],
"metadata": {
"version": "0.1.2",
"license": "Apache-2.0",
"description": "XPath grammar for tree-sitter",
"authors": [
{
"name": "getodk"
}
],
"links": {
"repository": "https://github.com/getodk/web-forms"
}
},
"bindings": {
"c": true,
"go": false,
"node": true,
"python": false,
"rust": true,
"swift": false
}
}
14 changes: 8 additions & 6 deletions packages/tree-sitter-xpath/scripts/build/parser-types.mjs
Original file line number Diff line number Diff line change
@@ -1,16 +1,19 @@
// @ts-check

import { spawnSync } from 'node:child_process';
import { mkdirSync, writeFileSync } from 'node:fs';
import { writeFileSync } from 'node:fs';
import { resolve as resolvePath } from 'node:path';

const CWD = process.cwd();
const distPath = resolvePath(CWD, 'dist');
const baseTypeGeneratorPath = resolvePath(
CWD,
'../../node_modules/@asgerf/dts-tree-sitter/build/src/index.js'
);

const baseGeneratedTypes = spawnSync('node', [baseTypeGeneratorPath, CWD]).stdout.toString('utf-8');
const baseGeneratedTypes = spawnSync('node', [baseTypeGeneratorPath, distPath]).stdout.toString(
'utf-8'
);

const correctedTypes = baseGeneratedTypes
// Replace broken interface constructor signatures with constructor declarations
Expand All @@ -30,7 +33,6 @@ const wrappedTypes = [
'',
].join('\n');

mkdirSync('./types', {
recursive: true,
});
writeFileSync('./types/tree-sitter-xpath-parser.d.ts', wrappedTypes);
const outPath = resolvePath(distPath, 'tree-sitter-xpath-parser.d.ts');

writeFileSync(outPath, wrappedTypes);
Loading

0 comments on commit c5d5982

Please sign in to comment.