Over the last few months, we've been seeing some new tensions arise within our teams - some that are unique to engineering, and some that feel universal for groups of people working alongside AI.
My first attempt to put my finger on what was going on was by thinking of us, as engineers, as so many Wizards of Oz - each pulling massive levers, creating huge change with the near-unlimited agency afforded to us by AI. And, critically, each of us hidden behind our own little curtains: operating in increased isolation. With more focus on the controls, we think less about what's just beyond our field of vision - our team, what they're doing, and how it all fits together. This aligns with the pervasive industry narrative of the individual contributor (IC) AI wizard, delivering on a compelling equation of omniscience + agency = shipping fast.
Of course, at ustwo, we have very few omniscient IC AI wizards - we tend to work in teams, thinking about human problems together, in real space and time. So while my Wizard of Oz metaphor still feels like it holds water when thinking about what it feels like to be a developer today, it stops short of explaining the specific tensions we're seeing arise. When the individual's work is done, and is presented back to the team for review, we're seeing new points of friction:
- authorship ambiguity: our uncertainty about whose work we are responding to
- change magnitude: the impact of overwhelm on our ability to respond with detail and nuance
- completion deference: the feeling of near-inevitability when a new contribution seems to have arrived fully-formed
Each of these have high potential to create conflicts within our teams. Let's get into what we mean, and think about some new defences. If you or your teams are working with AI, I hope it provides some language to triangulate new tensions, and some possible remedies that help to ease them.
Note: these are stated in terms specific to engineering teams, but if you squint your eyes a bit, you can see them fitting squarely across non-engineering work as well.
Authorship ambiguity
Reliance on AI puts reviewers in a position where they can't assume a consistent logical underpinning to the work. This shifts the review from "does this look right?" to "I have to verify everything in here". E.g. if I know a human author, and know how they might approach a problem, the effort I exert in a review is to understand the broad strokes and to make sure the solution is complete. Whereas if I have little trust in the author (which we shouldn't with AI), I have to interrogate everything but the most banal boilerplate. Not knowing whether I disagree with your original thought, or if I disagree with something you have waved through as sufficient is taxing.
A potential solution to this could be clearly flagging original thought vs. generated work. In non-coding work, this could be an agreement that certain types of communications are always human-written, even if AI-inspired*.
This, of course, needs to be made safe - flagging parts of our work as AI-generated needs to be viewed as highlighting where effort should be concentrated, and as a courtesy, rather than a sign of half-baked work.
Left unaddressed (which I expect is more likely in teams under stress), this can lead to two negative outcomes: waving through work with insufficient review/pushback, or rejecting work perceived to be AI-generated on a wholesale basis, without evaluating its possible merits. The latter can even overlap into toggling reviewers into a "distrust mode", wherein they go out of their way to find fault in work (or, worse, all of the work of a particular colleague) because they believe it to be AI-generated.
Defences worth experimenting with:
- Adding lightweight provenance to PRs and other documentation to indicate if something was human authored vs. AI-drafted/human reviewed
- Making it psychologically safe to offer up that AI was involved in the creation of a work product. It is actually a courtesy - the new workplace faux pas should be passing off unreviewed, AI-generated work as anything but that.
Authorship ambiguity creates a tension of trust. But even when we're clear-eyed about authorship, AI introduces another issue for us to grapple with: scale and overwhelm.
Change magnitude
Much has been written (by us, even!) about how working with AI-assisted tools creates a natural tendency to ship reams - way more than normal human output. This creates a new bottleneck - the reviewer. When a PR becomes extra large, we struggle to keep the full context in our brains and provide thorough review. A worrying effect of this is that the sheer mass of the change subtly (though unintentionally) discourages detailed, surgical corrections - it feels like adding an Oxford comma in the middle of War and Peace.
The industry has shifted focus to solving for this new bottleneck in recent months, but the answer likely comes down to either a) relying on more AI to do the reviewing, b) YOLO, or c) forcing ourselves back to human-sized changes (with or without some combination of a and/or b). If AI-generated code requires more scrutiny (it does) and our natural tendency is to produce more of it, we need to curb that instinct. Otherwise, the volume of changes under review will overwhelm our capacity to do so effectively.
Defences worth experimenting with:
- Coding guidelines that set explicit guidance for maximum PR size
- AI usage guidelines that specify work is to be shipped incrementally, even if it could be completed in one shot
- Parity enforced between PRs and work items (like tickets on a board). Shipping a feature and some other things at the same time should be avoided.
Increased change magnitude is one way that increased output can have a negative impact on review. Let's talk about another, more subtle one: work that arrives seemingly complete, with enough polish to make it feel nearly inevitable.
Completion deference
This describes the unshakeable feeling that something of substance must have taken considerable effort. If an engineer produces a full-blown artefact (working code, or a system of related changes), human engineers will be naturally hesitant to burn the whole thing down and insist on a more incremental approach in the face of having something in-hand that might functionally work.
We have seen this pattern play out on recent projects: a team member with a desire to make a meaningful impact quickly produces a detailed, polished contribution, but fails to take into account human context that has been built up before they joined or outside their field of vision. Because of the perceived completeness of the work, the reviewer is left with a sense that they either need to capture a detailed critique of each incorrect point (a very large effort), defer to the deliverable as a whole, or reject the deliverable outright - and the latter feels mean. Particularly when such a deliverable is created by someone new to the team, its creation can read as a shot across the bow of the existing team members, even when this is not the intent - the very existence of work so complete infers a judgement on the part of the contributor that they know better than the rest of the team. This can leave engineers feeling pushed around by others with less context, wielding powerful AI exoskeletons that can inadvertently trample over nuanced knowledge built up within the team. Of course, our contributor was simply trying to be helpful; but the authority of a complete-looking deliverable puts their peers in a defensive position.
Defences worth experimenting with:
- Encourage team walkthroughs of larger artefacts before PRs are raised. This can invite necessary conversation and avoid the appearance of the author presuming a piece of work is complete/nearly complete before commentary from the team
- Allow "reject and redo it smaller" as an encouraged/normalised response to work under review. Making this response a part of business as usual will remove the perceived social cost that reviewers are fighting today
Completion deference is a close cousin of the sunk cost fallacy (we've already gotten this far, so we may as well carry on), but with a key difference: the sunk cost is actually an illusion; a holdover from just a few years ago, when output scaled linearly with human effort. Look out for it as you respond to others' work - it doesn't take too long working with AI-empowered colleagues for it to crop up.
A new class of human problems
These three phenomena show up independently, but when they all appear simultaneously, we are in real danger of missing the mark of quality that we aim for. If we're unsure who's done the work, we can't skim with a charitable eye. When the volume of changes is overwhelming, it's not feasible to spend the time we need and so we might not bother even suggesting small corrections. And when AI enables people to push complete-looking things on their peers to digest/review, the onus is placed on the reviewer to put their foot down in order to keep our quality up. While each of these three considerations are technical in nature, the problems they create are social - they are pushing our teams in ways that feel uncomfortable - because AI is a new, silent team member that we haven't fully acclimated to just yet.
Our challenge today is simple, if not easy: stay transparent with each other as the line between human work and AI work becomes increasingly blurred. Judging our new AI team members is not the point - but seeing them clearly and acknowledging that humans still hold a critical piece in delivering great work is more important now than ever before. Emerging from behind our respective curtains and speaking honestly with those building alongside us is our way through these new tensions.
* For better or for worse, every single word of this blog post was carefully chosen by a human being.