One of the challenges when picking up a new programming tool or package is moving from the very basic ‘Getting Started’ page to the vast array of API documentation. The middle ground is immense and disorienting. It takes effort and persistance to advance — to actually learn the technology. The cognitive load is heavy and the path unmarked with only trace evidence that a sojourner has crossed these paths before…the evidence hinted at in Stackoverflow questions or blogs — most only half written. I suspect this is one of the reasons there is so much movement across different technologies — maybe Trello will solve my problem? Or Notion? Or…? Well, a pencil and paper will do just fine, as will Notepad. The challenge is the cognitive effort required to move from the adoption of a tool to its mastery.
Recall the last time you picked up something new — in my own memory, this might be uv or polars. Both of these have nice ‘getting started’ or ‘quickstart’ pages so that the data can quickly be loaded and operations performed. Okay, but now I want to do a specific task with my data. How can I accomplish that? The expert (see many such instances on Stackoverflow) will wonder why the novice didn’t just consult the documentation? Didn’t you see that polars has a DataFrame.concat function that does just what you needed it to? Why can’t you just read?
I don’t intend to absolve all such questioners of laziness — many do just try to avoid the cognitive challenge of actually having to think. But others have put in the work, and still can’t figure it out. Their question just requires the slightest of pointers in the right direction. Often, this is a simple transformation of their perspective (i.e., the perspective from which a tool is written) as with someone moving from one language (e.g., SQL) to another (e.g., pandas or polars, or R). The question is often phrased (perhaps implicitly) as ‘how do I use a where clause in polars‘? While the answer might be filter, it is more transformative for the SQL proficianado to adopt polars or pandas thinking. This is what is really happening in the ‘middle ground’ and it takes a great deal of effort and practice.
Another example has been my own adoption of uv from pip. The paradigm of pip is to install packages and record those in a requirements.txt file or (better) pyproject.toml under the dependencies. uv shifts this paradigm: now we operate from the perspective of a lockfile. The lockfile represents what should be installed when ‘syncing’. From a pip perspective, what’s currently installed is basic. This reorientation is always obvious in retrospect, but transformational when the ‘aha’ moment comes.
You can see numerous examples of the failure to move from ‘novice’ to ‘professional’ as someone explores and writes about some new tool or package. For example, consider the innumerable pages on Paperpile which just rehash Quickstart or Tutorial pages (I tend to avoid such sites unless I’m particularly desperate). Many blogs do the same and fail to challenge themselves advancing into the great unknown — perhaps they’ll do it wrong? Perhaps they’ll make a mistake? Ay, here be dragons, and hours of toil and despair. Yet our hero can emerge at the far end with his conquest (i.e., knowledge or the dragon’s gold, depending on how far you want to take the analogy).
As I have suggested elsewhere, chatbots can certainly help in this endeavour so long as they are used appropriately. Not as replacements for thinking, but as guides to speed along the process. The question of ‘how do I use the where clause in polars‘ will give an instant response with an example. Sure, you could give it a more specific problem statement — but, really, use these as a guide to learning.
There is another side to this as well, and that lands in the lap of the core developers (or core documenters, as the case may be). The challenge of documentation is that it’s difficult to write because the writer must adopt the perspective of the neophyte, and use the language of someone who’s barely grasping the current concept, providing them with the next step. But this is hard. Not only is the next step unclear (due to the nearly infinite possible next steps), but to explain the concepts in a way that the learner can actually understand. The heart of this matter is the ability of the core documenter to communicate well. A response of ‘why didn’t you just read the documentation?’ may be inappropriate as the user encountered a mental jump. (NB: I do understand that many are just trying to get an answer and avoid the learning, but hopefully chatbots will help to filter these individuals out.) These could be perceived as documentation bug reports, where the proper response is to revisit the documentation and help overcome that gap.
The next time you receive a question which seems obvious, instead of providing an answer, write a new case example or better explanation in the documentation and see if that helps. The questioner will thereby help to test your documentation and improve the overall comprehensibility of the documentation.