"Please don't pick the most provocative thing in an article or post to complain about in the thread. Find something interesting to respond to instead."
I've taken the provocative bit out of the title above, but please remember that we want reflective comments, not reflexive ones, in HN threads.
Really glad to see folks looking into quantitative approaches to give agents feedback on code quality. This post looks like a good start!
My main feedback for the authors would be, the most important problems for sloppiness are global properties, not local ones. In my experience an agent, like a human, has finite capacity for its attention, but if it runs into local sloppiness that gets in its way, it can fix it on a by-need basis. The technical debt issues that matter are usually global issues that aren't so easy to fix: they require global analysis and global refactoring.
I don't know the answer, but I think we're going to need ways to measure architectural properties, like separation of concerns, clear architectural layering, well-defined interfaces, etc.
Coding is not just the program running in memory, its also the process of distributing the mental model of understanding among the team.
If humans increasingly are kept out of coding, then who holds the mental model?
If AI holds the mental model, by definition human prompts will be over lossy channel. This is true without AI too. Software quality is directly dependent on good devs that translate from business/PM speak to technical decisions.
So is coding solved now? it was already solved decades ago.
We'll invent other tools to help with the mental models, ones that are more intuitive.
Imagine being able to walk into a house and see immediately the leaks dripping from the ceiling, the clunky layout and smell the stinky garbage that hasn't been collected.
We're already in a sort of IDE when you think about it, we're operating a much larger mental model than any code-base, and we rarely remember that we're operating with such a model.
> Imagine being able to walk into a house and see immediately the leaks dripping from the ceiling, the clunky layout and smell the stinky garbage that hasn't been collected.
No need to imagine. Just open the README.md of many projects.
Mental models existed before coding. The pyramids were not built by just a couple of guys hauling some rocks together. It was a highly structured organization.
Or the East India Company or many other great organizations that achieved big things before computers.
Imho a better way of thinking about it is in terms of views - you can have an unimaginably complex codebase that nobody reads in full, but either agents or humans receive views/perspectives on that codebase.
Before LLMs we couldn’t have built tools that would deliver such views, now we can.
I mean... Nobody NEEDs to hold a mental model, but we've been increasingly trying to build that mental model into an agentic codebase... maybe we're doing it wrong but it is NOT going well.
The agents jot down absolutely everything (and more, which is the issue), thousands and thousands files of perceived business logic, decisions, workflows, architecture, data modeling, etc... It gets stale extremely quickly - we change one thing in the system and 570 "shards" need to be updated. I stumble on more and more things every day that are untrue about the agents "mental model" and it appears to just keep doing it. One small addition not properly reviewed by a human introduces a cascade of misconceptions system wide.
We've tried these "views" you mention, we call them subsections, but it runs into the same problems. One falsification ends up poisoning entire projects.
Remember when we use to refactor functions in a codebase in order to slim down complexity? Yeah - we ended up doing that but with documentation... IMO code is deterministic and we have trained professionals who know how to read it quickly and modify it. (We don't even need to write it anymore, but reading it is the simplest way to ensure no-loss understanding of a piece of functionality.)
Oh, I just dropped all the comments. Memory and session histories are for storing these things, comments just add confusion.
Subsections - how do they run into the same problem? What falsification? The views I mean are dynamic ad hoc. Every single small task gets its own temporary view.
But perhaps I’m misunderstanding something that you’re saying?
The only difference with the past that you need less time to dig through the codebase or documentation, the agent can do it for you and provide only meaningful info, but without a "Mental model"(or knowing what's going under the hood) your own prompts will be the main reason of the sloppiness and high token usage.
Having reached the same conclusions as the author led me to create my first agent to do architecture review, and that's how I learned about the metrics behind good practices that I'd been following for years. LCOM, cyclomatic complexity, that kind of stuff...
It's so easy to ship a lot of code, more effort should be put into ensuring the code is correct, with self-improving feedback loops that involve developers, and dedicated tooling...
But again, a while ago, everything was about prompt engineering, and now you can express you idea vaguely and get a somewhat working result, so this likely will evolve fast as well...
"Please don't pick the most provocative thing in an article or post to complain about in the thread. Find something interesting to respond to instead."
I've taken the provocative bit out of the title above, but please remember that we want reflective comments, not reflexive ones, in HN threads.
https://hn.algolia.com/?dateRange=all&page=0&prefix=true&sor....