Eight hours with an LLM, thirty minutes without

Today was somewhat of a wasted day, with some reinforcement learning.

This might be a bit different from everyone else's experience with LLMs for coding. People around me are shocked, and question it, when I tell them I don't use LLMs to write code. Most of that comes from past experiments to gauge the cost, time and benefit for the kind of work I do. Some of it has to do with IP rights, too.

The experiment

Today was one of those experiment days. I picked a fairly standard problem and hoped an LLM would make it go faster.

The problem was passkey attestation. Passkeys are one of the phishing-resistant authentication methods, and you can't skip them when building an identity provider. The goal was a program that regularly downloads the FIDO Metadata Service (MDS) authenticator metadata, parses its attributes, stores them, and uses them to validate attestations. Enterprises love this feature for security reasons.

Doing it by hand, I already knew where to download it from, how to parse it and what to do with it. For the LLM, I had to prepare all the context: scrape the references, links and metadata to feed in as input. It's the difference between doing it yourself and explaining it to someone else.

Fourteen files later

After multiple iterations and a bug in certificate validation, it generated something that worked, spread across 14 files. It took a good few hours. Time flies when you're hoping someone else will do the work and all you do is interact, wait and run the program. (LLMs seem much friendlier with Python; it complained about the lack of a Java environment.)

The data was showing up, but something didn't make sense. One attribute is supposed to say whether a credential is multi-device, meaning whether the passkey syncs across devices. Even though I had told it explicitly which field to parse and how to interpret it, the code returned false every time.

It wrote ten lines of comments explaining exactly how to parse the field, and then didn't parse it that way.

Ironically, those comments read well at a high level, so I didn't check the details. Part of the reason was the sheer volume of code: I was busy verifying the critical parts, like certificate validation, and took the simple ones for granted based on the comments. Other parts of the parsing were off too.

The result

After investing eight hours, I deleted all of it and wrote the same flow in about 30 minutes, in just two files 😊

Disclaimer: I have more than a decade of utility functions and files that I reuse right away, which saves me quite a lot of time.