Tl;Dr: Ship the minimal viable code with the package, implement a default skill, have your coding agent answer questions about its own abilities and implementation in depth. Never update docs again.
I, like most engineers, always found it a chore to update the docs in sync with code changes. Especially now that code change velocity is so high that the likelihood of your docs being obsolete within a week from previous is no longer insignificant. It has become somewhat of a Sisyphean task.
One of the feedbacks I heard from early adopters was the lack of clear documentation, and they are right. If you check dirac.run/docs, the most appropriate description that comes to mind is ‘embarrassingly low on details’.
While I have been aware of the limitation, updating docs always felt like an important but less urgent task when there are 10 higher priority tasks on your plate. Below is my own chain-of-thought reasoning (dramatized, but only a little).
Users are asking for updated docs… what can I do?
Update docs manually? Nah that’s just too time consuming to get it right and is a neverending task.
Why solve at all when even the largest multi billion dollar companies’ docs are also often outdated? This, while true, feels like the broken windows effect applied to software and it'll make the agent look less credible.
Get the LLMs to do it for me? LLM does a first pass to completely read the codebase and write docs. Subsequently, it reads the code updates and keeps updating the docs. This works to an extent but in the first phase, you need manual involvement, and in subsequent phases, you need LLMs to continuously watch the commits, find relevant parts and update the docs.
The problem? The LLMs are overeager to act and sometimes update similar but not exact things that need updating. This will get worse as the total doc size grows. Furthermore, not all users are on the same version of your app, and docs only address a single version.
I truly do not want to spend any time updating docs, if I say otherwise, that would be a lie. What is the laziest solution I can think of?
I need something that looks at the codebase and answers the user’s questions that way I manually would. If only there was a mechanism that can read code and answer constrained questions in natural language! Oh wait, isn’t that like ‘skills’ supported by the exact fucking thing I am building? Plus code is open-source anyways so I can easily get it to read github to answer questions.
Isn’t github going to add latency and potentially block some requests, causing a bad user experience? Yeah that’s right, we can’t be sure.
How about shipping minimal functional source with every release, and creating a built in slash command /askDirac that will just point Dirac to that source code to answer specific questions? Plus, there is an added benefit of user’s getting answers from the exact version they are running. Hmm, that might work, let’s try.
Act mode
The source code was around 7MB, which is not insignificant but not something anyone would mind either.
Made updates to packaging process so the relevant part of the code ships, added the new slash command. I worked for about two hours to get the first version ready. Shipped it soon after.
Result
Analysis
Benefits:
- Users get the answer as opposed to having to find and read the docs and connect those to their questions
- users get the precise answer tailored to the version they are using, they never have to leave their dev setup.
- Enables users to open well-informed issues, and contribute PRs easily
Downside:
Costs a few cents of inference per use, but feature clarification is infrequent enough and saves many minutes of human time digging though the docs (and frustration when docs are absent) or posting on github that it makes it a net benefit.