robots: User-agent: * might appear twice in robots.txt

[davidsneighbour]

Given that users could configure for instance:

[[dnb.robots.useragents]]
name = "*"
disallow = ["/cdn-cgi/"]

This will lead to two separate User-agent: * sections. One when the local configuration is parsed and one when the global disallow via frontmatter is ranged.

Fix: check in the first range for name and if it's * move the items to the global range.

[davidsneighbour]

This requires a system to collect user agent data.

Overriding in this order:

• config.params
• section params
• post params