More Docs

[jamesturk]

• pros and cons
• usage / cookbook
• cool ghost

Pros

• nearly free in terms of time/energy to see if it works or not, throw it at a problem and see if the results are high enough quality, might need to write one or two CSS selectors
• great for few pages where content changes frequently and scrapers might otherwise be too expensive to maintain
• possibility of random errors that are hard to detect, should not be used without validation if results matter

Cons

• scrapers will be dependent upon OpenAI until other models are available/equally good
• every run costs something, even if pages don't change at all
• not good for scrapers with lots of pages & lots of runs
• bad at list pages due to context size issues and speed

[jamesturk]

Examples of hooking into various parts of process:

• URL -> HTML (can pass in any HTML)
• HTML -> clean HTML (preprocessors)
• API Calls: not possible yet, Make API backend pluggable to allow for non-OpenAI models #18
• JSON validation & refinement (postprocessors)