AI slop got better, so now maintainers have more work

Once AI bug reports become plausible, someone still has to verify them

If AI does more of the work but humans still have to check it, you need more reviewers. Now that AI models have gotten better at writing and evaluating code, open-source projects find themselves overwhelmed with the too-good-to-ignore output.

For the curl project, that has meant less AI slop and more demand upon maintainers who have to evaluate more plausible vulnerability reports.

"Over the last few months, we have stopped getting AI slop security reports in the curl project," said Daniel Stenberg, founder and lead developer of curl, in a social media post. "They're gone. Instead, we get an ever-increasing amount of really good security reports, almost all done with the help of AI."

The reports, said Stenberg, are being submitted faster than ever before and are imposing a growing workload on maintainers.

According to Stenberg, the situation is similar for other open source maintainers.

Linux kernel maintainer Greg Kroah-Hartman recently noted how AI-assisted bug reports contained less slop and more valid concerns. He said that the Linux team has been trying to deal with the increased volume, but implied that smaller teams might be struggling.

Even if the reports are better, the issues being identified aren't necessarily security flaws that can be exploited and need to be corrected. As evidence, Stenberg points to curl's public list of closed reports. Most of the reports have been closed because the issue isn't a serious threat, even if it might be something worth correcting. 

For example, a data race in a curl library was initially discussed as an issue that might get a CVE. But it was eventually fixed in a pull request, with the bug deemed to be simply "informative."

Stenberg, back in 2024, called out the problem of AI slop bug reports and, earlier this year, went so far as to stop paying awards for curl vulnerability reports. His goal was to remove the incentive to submit erroneous or unsubstantiated reports, whether those came from automated systems designed to maximize financial gain while minimizing effort or from people using AI tools who shirked their obligation to check the AI's work.

Other organizations have taken similar steps, most recently the Internet Bug Bounty program, which said it would stop issuing monetary awards for vulnerabilities at the end of March.

"The discovery landscape is changing," the program maintainers said in an announcement that also shuttered the Node.js vulnerability award program. "AI-assisted research is expanding vulnerability discovery across the ecosystem, increasing both coverage and speed. The balance between findings and remediation capacity in open source has substantively shifted. We have a responsibility to the community to ensure this program effectively accomplishes its ambitious dual purpose: discovery and remediation. Accordingly, we are pausing submissions while we consider the structure and incentives needed to further these goals."

Linux maintainer Willy Tarreau responded to Stenberg's post by noting that the Linux kernel team has had a similar experience to those working on curl. He argues that more needs to be asked of those making bug reports.

"It's time to update the reporting rules to reduce the overhead by making the LLM+reporter do a larger share of the work to reduce the time spent triaging," he said.

Capable AI tooling doesn't increase the capabilities of the humans in the loop. Much of the notional productivity gain from AI may just be AI tool users moving the cost of code review off the books. ®