Configuring Neovim treesitter from first principles

In April I wrote a migration guide for updating the nvim-treesitter plugin to Neovim 0.12. Ironically enough, a day after that was posted, nvim-treesitter's maintainer archived it.

While this plugin still works, seeing it break my setup after the 0.12 update made me curious about what's going on under the hood. Luckily, Neovim 0.12 ships almost everything built-in, so you and I can run treesitter with no plugin at all.

nvim-treesitter is a manager sitting on top of three layers: parsers, queries, and a bit of Lua to glue it all together. Let's dive in.

Parsers

A parser is a small compiled library, one per language, that turns source text into a syntax tree: lua.so, python.so, and so on. (A syntax tree, or formally a parse tree, is a data structure that represents how a program or snippet is organized.)

Neovim 0.12 only bundles support for seven languages out of the box: c, lua, markdown, markdown_inline, query, vim, and vimdoc. For any other language, you need to compile the parser yourself.

First, find the grammar repo for your desired language. This requires a quick Google search, like <lang> treesitter grammar. This "grammar" repo contains instructions and rules on how to turn a source text in that language into a syntax tree.

Next, install the tree-sitter CLI, clone the language's grammar repo, and build it. Below is JSON as an example, but the pattern is the same for any language: clone tree-sitter-<lang> and build it to <lang>.so.

git clone https://github.com/tree-sitter/tree-sitter-json
cd tree-sitter-json
tree-sitter build -o ~/.config/nvim/parser/json.so

Neovim searches for parsers as parser/{lang}.* in every directory on its runtimepath, so dropping json.so into ~/.config/nvim/parser/ is all you need to do.

Queries

Once we have the syntax tree, treesitter can run queries in order to do jobs like highlighting, folding, etc. A query is written as an S-expression pattern (sort of like regex but for the syntax tree instead of raw text) that matches the right parts of the tree for a given job.

Queries also assign matches with capture names, which are labels that look like @keyword, @string, or @comment. Neovim uses these labels to decide what should be highlighted, folded, or otherwise processed.

Queries live as .scm files under a queries/<lang>/ directory on the runtimepath, one file per job:

• queries/<lang>/highlights.scm for highlighting
• queries/<lang>/folds.scm for folding
• queries/<lang>/injections.scm for embedded languages

For the seven built-in languages, Neovim already bundles their queries along with the parsers. For anything else, drop your own into ~/.config/nvim/queries/<lang>/.

The big query corpus that nvim-treesitter shipped for hundreds of languages lives in its repo, so you can copy the ones you need into your config. Note that your file replaces the bundled queries for that language (first on runtimepath wins), so prepend ; extends to it if you mean to add to them instead.

Putting it together

Great work making it here. Let's run this to start highlighting:

vim.treesitter.start()

With no arguments, treesitter attaches the highlighter to the current buffer using the language for its filetype. It also turns off the old regex syntax highlighting. Re-enable that with vim.bo.syntax = 'ON' if some plugin still needs it.

To auto run this everywhere, put the call in a FileType autocommand:

vim.api.nvim_create_autocmd('FileType', {
  callback = function(args)
    pcall(vim.treesitter.start, args.buf)
  end,
})

The pcall swallows the error for filetypes with no parser installed. For folding, set two options:

vim.wo.foldmethod = 'expr'
vim.wo.foldexpr = 'v:lua.vim.treesitter.foldexpr()'

Unfortunately, Neovim doesn't provide indentation out of the box. To set it up, find it in the archived nvim-treesitter code, although note that it's still experimental. For most languages, I think Neovim's built-in indent rules are good enough.

When to skip the plugin

Just because you can doesn't mean you should. When going plugin-free, you give up things like :TSInstall, automatic parser updates, the giant query corpus, and textobjects like vaf to select a function. If you use those, keep the plugin for convenience. Its main branch still does this job well and now calls these same core APIs under the hood.

But by setting up treesitter yourself, you get something with no moving parts that can break: no plugin rewrite forcing a migration, the whole thing is a parser file, a few queries, and a small snippet of Lua.