Yesterday, I gave Codex a chalk talk while out for a walk. There was no chalkboard, just a voice conversation about managing kidney cancer treatment toxicity. Codex turned it into a presentation I’m going to give to nurses and pharmacists.
It was the first time I’d worked this way. I was using OpenAI’s Codex in live voice mode, teaching it how I wanted the presentation put together. It felt like working with a student, and I really enjoyed it.
At one point, we were discussing liver toxicity from immunotherapy. Codex proposed prednisone followed by infliximab. I corrected it: for immune-mediated hepatitis that had not responded adequately to corticosteroids, the guidance we were discussing pointed to mycophenolate mofetil. Infliximab is used for some other immunotherapy toxicities, but the Society for Immunotherapy of Cancer guideline advises against it in liver injury because it can itself cause liver damage.
Once I explained the distinction, Codex found the supporting information in the review we were using. It hadn’t applied that distinction in its initial answer.
Explaining the correction was useful for me too. Codex came back with specifics from the paper, reinforcing what I already knew and adding a few practical points I picked up as we worked through it together. I was contributing clinical knowledge, and the conversation was helping me revisit and extend it.
That experience gives me a way into a question researchers are studying: what do we learn while completing work with these tools?
What stayed after the task was finished?
In the research preprint How AI Impacts Skill Formation, Judy Hanwen Shen and Alex Tamkin studied developers learning an unfamiliar Python library called Trio, which helps coordinate tasks running concurrently. The main randomized study included 52 participants. One group could use a GPT-4o assistant; the other worked without it.
Participants completed two coding tasks, then took a quiz without assistance. The quiz tested their understanding of concepts, their ability to read code, and their ability to identify and diagnose errors. That distinction matters: the researchers examined what participants understood afterwards, beyond whether they had produced working code.
The assisted group scored significantly worse. An exploratory breakdown found the largest gap in debugging. There was no statistically significant difference in average completion time.
This was a short experiment involving an unfamiliar programming tool. It did not establish long-term skill loss, or what happens when an experienced physician uses voice conversation to prepare teaching material. But testing people after assistance was removed made it possible to examine something the completed work alone could not reveal.
The conversations differed
The researchers also reviewed recordings of how participants worked. Their account of those interactions is particularly useful.
Some participants handed over the coding. Others increasingly relied on the assistant, or repeatedly asked it to fix errors. Those patterns were associated with lower quiz scores.
Among the higher-scoring patterns were people who requested explanations alongside generated code, asked follow-up questions to understand it, or asked conceptual questions and then wrote the code themselves. Receiving generated code did not necessarily mean someone stopped trying to understand it.
These were small groups identified through qualitative analysis. Participants were not randomly assigned to different conversational habits, so the findings do not prove that asking for explanations caused better learning. Nor did the immediate quiz establish how much anyone retained months later.
Still, the findings suggest something worth examining in our own use: when we ask a follow-up question, are we trying to understand the work, or asking the assistant to finish another part of it?
Both can be useful. They serve different purposes.
I knew where to interrupt
During my walk, I was explaining how to teach material I already knew something about. When Codex proposed the wrong next treatment for liver toxicity, I could recognise the problem and redirect the conversation. Working through the source afterwards reinforced that knowledge and added detail.
I can describe that as a valuable learning experience without claiming I have measured how much more I will remember. I also would not expect the same exchange to work equally well for everyone.
Someone encountering immunotherapy toxicity for the first time might not know where to interrupt. They could ask for an explanation and receive one that sounded reasonable. The higher scores among some participants who worked through explanations are encouraging, but they do not settle how a learner should recognise an explanation that needs challenging.
That leaves me with a practical question about teaching. If we want people to learn while using these tools, what opportunities do we give them to explain a decision themselves, check it against the source, and work through a correction? Looking only at their finished presentation would miss much of that.
I enjoyed giving Codex a chalk talk. I got a walk, a presentation, and a useful discussion of the material. The harder question is how to make that kind of learning available to someone who is still acquiring the knowledge I brought to the conversation.
And when Codex presents me with information in an area I know less about, how will I recognise what I’m missing?