Better LLM assistance for Statamic and Antlers syntax?

[jensolafkoch]

I’m curious about your experiences with current LLMs like Claude 3.5 Sonnet or GPT-4o in relation to Statamic and its Antlers syntax, particularly when it comes to tags, modifiers, and inline PHP.

It seems that Statamic-specific material is sparse in the training data, as one might expect. Do you know of any models that are better suited or specifically trained for Statamic and Antlers? Alternatively, are you aware of any initiatives aimed at fine-tuning LLMs for this purpose?

There’s always the option of injecting such knowledge into the context manually—has anyone explored or shared their experiences with this approach?

BTW, I mostly use ChatGPT Plus and Codeium in PhpStorm, as well as experimenting with the new and promising Windsurf editor from Codeium.

[edalzell]

We tried to feed the Statamic docs into an LLM, but produced wrong results very often, so we stopped trying.

What have you tried searching for in our docs that you haven't found?

[jensolafkoch]

It’s not about not finding information in the docs—I do. 🙂

I’m looking for a way to get better LLM results when dealing with Antlers. Automated workflows, especially in Codeium Windsurf, are quite powerful, at least for PHP/Laravel and broadly used tech stacks, but not for Antlers.

I tried today to give the LLM a system message to always check the docs (the locally cloned docs repo, that is) before generating Antlers code, but the results are suboptimal, to say the least. 🙂

In what way did you try to feed the Statamic docs? As full context, as instructions to search the docs online, or via fine-tuning a model?

[jensolafkoch]

I settled for an approach in the Windsurf editor (VS Code clone) from Codeium (with its long stunning reasoning workflows), I set up a global rule containing the following block (besides a lot of other details re. my dev environment and preferences, including project rules for source and target and reference projects):

## Authoritative sources: Compulsory reading at the beginning of each chat
- Always read the lists of all Statamic modifiers and tags at the beginning of a chat:
  d:\dev\apps\stat-docs\_stat-docs. Confirm this with: “Modifiers and tags read.”. Use only these in Antlers code!  
- Always read the file d:\dev\apps\stat-docs\_stat-docs\Antlers-Book\toc.xhtml at the beginning of a chat. Confirm this with: “ToC read.”. This table of contents refers to the Antlers Book in XHTML format (EPUB structure) located in the same directory. Use the ToC to search for Antlers solutions if needed. I will simply refer to “the book” in prompts.
- Always read the filenames of the documentation pages for individual modifiers and tags at the beginning of a chat, which are located in d:\dev\apps\stat-docs\content\collections. Confirm this with: “Statamic-Docs: Filenames read.”.

I cloned the Statamic docs from Github so that the editor can access all current docs via the collection contet files.

I also added an overview MD file with all current modifier and tag names.

And I extracted the content of John Kosters Antlers book (EPUB version) so that the editor (that si: the used models) can read the XHMTL files (PDFs don't work). (Of course I only use it with a don't-use-for-training-LLM access ...)

With this setup I can always say (see *) my editor to check any possibly hallucinated code again (and that's the case with Antlers a lot usually, with Claude 3.5 Sonnet or GPT-4o at least), although the clear prompting in the global rules to always check against the authoritative sources seems to help a lot.

(*) I nowadays always talk to my editor because for longer prompting that is sooooo much faster than typing. (As a Windows user I rely on SpeechPulse, a very interesting app which can use a GPU to locally use a whisper model).

As long as we don't have fine-tuned models which have been post-trained with this kind of information I believe this is the most effective way to go. I would be happy to hear any other ideas and experiences and possibly better approaches!