feat: nitro hook robots:init and robots:robots-txt:input

[harlan-zw]

🔗 Linked issue

Further work in improving #233

❓ Type of change

• 📖 Documentation (updates to the documentation or readme)
• 🐞 Bug fix (a non-breaking change that fixes an issue)
• 👌 Enhancement (improving an existing functionality)
• ✨ New feature (a non-breaking change that adds functionality)
• 🧹 Chore (updates to the build process or auxiliary tools and libraries)
• ⚠️ Breaking change (fix or feature that would cause existing functionality to change)

📚 Description

We need more granular hooks where we can have more preditable input / output without having to renormalize input that is already normalized.

[silverbackdan]

Thanks for cracking on with this. Looking at the comments in the readme perhaps I was actually trying to use this differently than planned.

So I think it'd be good to think about how I would be recommended to use it for my purposes.

As far as I understand, robots:init will be used for build time.

robots:robots-txt:input for runtime.

This is pretty good, and in my case, my module requires some default groups are added to prevent robots accessing CMS pages.

But then I am adding website user abilities to configure to some of the robot options through a website settings page.

1. Allow/block AI bots
2. Allow/block scrapers
3. Insert custom text

Therefore I'd expect to put this configuration into the new hook currently named robots:robots-txt:input

The consideration I'd have is when sitemap needs this information it calls upon getPathRobotConfig
https://github.com/nuxt-modules/robots/blob/main/src/runtime/server/composables/getPathRobotConfig.ts

I think while we are here it'd be good to configure how this composable can be called consistently and respecting whatever is done in the runtime hook. I think in the current setup that this may not happen.

I might be wrong, but this could be a reason why my sitemap was working when I deployed and then failed later after a few hours sometimes.

I've got a chunk of work right now but if I can jump in on some tasks on this PR I will - so long as we are coordinated on what to work on so we don't double up or conflict code :)

[silverbackdan]

Sorry, happy to revert, just seeing if there was a workaround - feels like the Nitro plugins aren't mapped properly from the Nuxt test-utils? Was just curious when I saw this and had 5 mins to see if there were other ways we could map to the plugin in the fixture