Autonomous AI Engineering

Three years ago, Retrieval Augmented Generation (RAG) was the “hottest thing” and within 12 months it was seen as outdated by many. Not that the grounding provided by RAG was faulty but the “single search-generate pass” was seen as too trivial for many questions. By that same notion, we have seen the very rapid rise of coding assistants which, in theory, help improve developer productivity. Many have now asked though, do we seek to improve individual developer productivity or should we be looking at methodologies which improve team productivity. Adding to that, Federal teams are constantly asked to do more with less, leading to debates about whether to focus on individual or team productivity. Here we look at Cognition, a second generation autonomous AI engineer and a concrete example of how it could help the Australian Federal ICT market.

About Cognition

Two years ago, Scott Wu announced Cognition with the flagship product Devin. Devin had lofty goals: instead of improving an individual developer’s loop (prompt cycles) to improving the entire SDLC. The first version of Devin had mixed results: some loved it, some hated it. Part of this was due to Devin and part was due to the capabilities of the models.

Since then, Devin and the models it uses have improved … a lot:

1. Shared Understanding: The concept of “Knowledge” of a codebase has evolved into “DeepWiki”, as a mature concept that understands both the macro and micro concepts of a codebase, understanding deep context from code and commits.
2. Better Prompting: Improved concepts from DeepWiki are used to help inform the prompting.
3. Automated testing: Being able to bring up a browser in a Windows or Ubuntu VM goes far to recreate the developer experience. Add to that the use of an LLM to “validate” outputs can shortcut the entire testing experience.
4. Reviews: In addition to creating the PR, Devin can run an analysis of the commit, looking for bugs, flagging issues and deep security problems.
5. Model METR: The amount of time a model can run to “replace” a human equivalent task before needing interruption has doubled every few months, from seconds to hours. This takes Devin closer to operating as a true asynchronous team member rather than just an interactive assistant.

Another aspect is the evolution of the Overton Window in development teams: We are recognising more and more that a shared team understanding needs more than a SKILLS.md - we need to be able to collaborate within sessions. A product manager or architect should be able to collaborate with an engineer. Concepts shown in DeepWiki should be a team asset as opposed to the individual learnings of a developer session.

And finally, from a Security Standpoint, we’re starting to grapple with concepts like whether the DevBox should be on the developer’s local machine. Plugging in an LLM with security API keys beyond the needs of least privilege creates a weak barrier to a potential data breach - which is especially significant in an Australian Federal landscape.

A Real Test

Devin promises to help with asynchronous development - I recently put this to the test. Whilst flying internationally with the spotty Internet provided, I decided to see if I could modernise an older GitHub repo I worked on: https://github.com/davidglevy/example-nifi-csv-cleaner

What was particularly gnarly about this one:

1. It was a NiFi processor, I hadn’t loaded up NiFi in years nor had my old developer setup - which previously would have needed NiFi to be installed. On a plane this was already prohibitive as the installation was 3GB
2. In the 7 years since I touched this, NiFi had moved on, from 1.x to 2.x
3. Although I had a basic unit test, I didn’t really have anything resembling a true integration or system test

The Goal

I gave Devin a number of tasks, in order, as I flew over the pacific:

1. Add a better README.md
2. Upgrade to a newer version of the 1.x NiFi (non-breaking change)
3. Run the processor in a system test and add a screenshot to the README.
4. Major upgrade: Upgrade NiFi to 2.x as a major breaking change
5. Re-run the processor in a new system test and update the README with a newer screenshot.

As I passed across the tasks from my flight, the innovation loop became infectious. Devin was exercising the full SDLC, and all I needed to do was check the comments and PRs. This is the cornerstone of responsible AI: the agent does the heavy lifting, but the human retains the final authority, reviews the code, and merges the PR. It’s about human-in-the-loop, not replacing the human entirely.

Maintenance: Ongoing Patching and Security Checks

This was a real eye opener, not only could we use Devin to perform major breaking upgrades and test them, but we could easily position this as an ongoing activity via the Automations feature. This by itself provides a massive boost for the Australian Federal ICT sector - we can now setup a monitor to look for new versions. This is further amplified by Devin’s new Security component, which can perform scans for vulnerabilities and similarly pass down the work for fixing.

Close

We have all seen the productivity boost for individual developers from first-generation copilot tools. But the next wave of Autonomous AI Engineering shifts the equation from individual contribution to team-wide delivery across the SDLC. For the Australian Federal ICT sector, where security, compliance, and resourcing are constant balancing acts, automating the most tedious parts of the SDLC improves operational resilience. Using tools like Devin to continuously patch, upgrade, and test legacy systems while humans retain architectural control and final approval, we can finally tackle technical debt at scale.