Okay, I'm Vibing
From net neutral to net positive ROI with LLMs
This post from my friend Aaron prompted (who is the LLM now?) me to take a beat on where I am personally with LLMs.
In a previous post I shared how I couldn’t tell if i’m more productive. I even made the case that productivity was an illusion, but things have changed. I have in the last three weeks checked in more code written primarily by Claude than in the previous 10 months combined.
Claude Opus has been a game changer for us and things are moving faster than expected. The change is not entirely due to the model alone. We are getting really good at leveraging it appropriately. Our team was able to produce a stunning CLI for Semantic layer management in record time. And, I was able to finish boring parts of the app (ie System settings and monitoring UI) faster than my most aggressive estimates. Another explanation of the speed here could be that both creating CLI’s and standard CRUD interfaces is pretty simple. But that is exactly the point. We are able to off load common app parts easily to the LLM. More complex features need more care and management but this definitely accelerates our pace.
What took me 2 years to build at Netflix is nearly feature parity in under a year with even more capability.
Greenfield Projects and LLMs
We are building a completely new app in fresh repo’s. I’m acting as a PM, UI Designer, Frontend Engineer, Backend Engineer, and architect all at once. This means context muddling for me and the agents. The temptation is to one shot entire features and call it done. The problem here is that one shotted code is high entropy. I’ve thrown away tons of one shotted features and handwritten them from scratch. The better approach for me is to segment usage of LLM by the role I’m performing rather than the feature I’m building.
What does this mean in practice?
First, I take a feature I want and spec it out completely from the end User experience POV. This is the Product Management role. I’ll take this spec and one shot it. Here I blindly accept all the code and don’t bother reviewing it. The goal is to figure out if the product spec is right. Does the UX make sense? Do we need sub features to better describe the main objective here? Or, is the feature completely wrong for our target use case. This is a game changer! You can protype features before even considering them for the final product. Throw away work like this could only be done with large teams and big money in the past.
From here, I’ll either iterate to test different experiences or start working on the architecture. To do that I’ll review how the LLM thought about it then re-architect it from first principles as it relates to the long term vision of our product. The routing paths, main files, code structure, and models will be outlined by me. Then I walk the LLM through implementing each in groups that make sense. For example, for precaching data lists, we have a model called Elemental and background job called ElementalJob. That whole implementation can be done together. Once the backend is done we move to the frontend and follow the same process.
This approach is easier for me and the agents to understand. Its step by step but 10x faster than what I alone could have done in the past.
Some will argue that I’m wasting time and I should just accept the one shotted code. I’m sorry I just can’t. I really care about the aesthetics of the entire app. From the frontend to the backend. I care what the URL looks like, what the class names are, and how the code is organized. I want it all to be aesthetically pleasing. And guess what? I can have it all now with the help of LLMs. This sort of thing would make me slower in the past but I can get it for free now.
Lastly, being focused on end to end aesthetics will pay dividends in the future. My code will be easier for agents and humans to understand and it will become faster and faster to add capability.