IDS AI Agent submissions - controls needed?

[andyward]

I appreciate the good intentions of what @BIMvoice and others are doing with AI agent driven workflows to help contribute to issues and bug-fixes in the BuildingSmart IDS spec & surrounding ecosystem. But I can't be the only one with a sudden increase in the number of Github notifications and Emails every day looping me in to an Issue/PR that I may have been involved in? I'm guessing the IfcOpenShell maintainers have the same challenge with their 100s of new PRs.

Claude etc create well-structured and (at least at a cursory level), well thought-out submissions. They can knock out a dozen PRs in the time it takes me to refill my coffee.

But when I'm spending time reviewing and providing feedback, who am I responding to? The agent or the human behind it? Do either deeply understand the decisions being made and impacts? Seems there's a mis-match between the amount of effort (CPU cycles) for an agent to generate solutions, comments, PRs and the human 'meat-bags' 😄 on the other end of the conversation to spend time understanding & reviewing the implications.

Some questions/concerns:

1. Should agents be proposing PRs on changes to a BuildingSmart standard? Allow numeric and boolean simpleValue (#342) #454
2. Should agents be making contributions /responses without a human in the loop? Add IFC2X3 occurrence/type mapping test case (#116) #458 (comment)
3. Should the maintainers be employing our own agents to sift the responses? (I don't think so, but asking the q)
4. Why don't we have a Contributing.md with guidance (or a Claude/Cursor.md)?
5. Why don't we have a PR template? - a simple question like "I have read CONTRIBUTING.md and both the human operator and any agent can explain every line of this change during review" - might help

As well as a Contributions.md etc, I think it would also be sensible to ask agents to rate limit their submissions to <4 a day (assuming they honor the instruction). We could also add some Github actions to gatekeep - (only accept PRs on issues with a certain tag/milestone?)

What's people's thoughts? Again appreciate Petru and co's great intentions in getting 'hands on' but it probably need harnessing if it's to be achieve the results we want.

[atomczak]

Hi @andyward, I agree with your concerns. My mailbox also exploded today, and new comments appear faster than I can read. We are in the process of reviewing the contributing policy at bSI, and will reflect that in the repo (contributing.md and PR template - good ideas).

Of course, there is also a risk of users disobeying the policies, and in that case we will have to limit the PRs to a narrow group of IDS contributors. Luckily, so far the intentions are only positive, and Petru is trying to improve IDS and resolve stale issues.

Following up on the IDS survey, we are looking into ways of reinstating the IDS coordinator and regular working group meetings.

[andyward]

My feeling is agents are probably better at following instructions than us humans. We just need to give them the rules/guard rails. When they start maliciously disobeying policies I think we've got bigger issues and I'll be off to my bunker 😄

Look forward to the survey results.

PS. May be worth taking a look at some of AI first repos (in IFC space maybe Ifc-Lite ?) for inspiration on setting up the guard rails. AGENTS.md etc.