tldr; software development is prone to distractions as new tasks spin out from the original task. Developers should try a productivity aid (just a notebook, text editor, or outliner to start) to help them manage this phenomenon, mainly by increasing their awareness of the issue as it happens.
For non-developers that might be reading this, the call stack (or just stack) is the place where an executing program keeps track of what it is doing. When part of a program calls to another part of the program, an entry is placed on the stack so that when the other part of the program is finished, the program returns back to the original point where the call was made, and the program continues on from there. This layering of the stack as one procedure calls another which calls another, and so on, can get very deep. The stack can get so deep in fact, that the execution of the program must be abandoned. When this happens it is called a stack overflow (which is where the name for the software development Q&A site comes from).
Why mention this (simplistic) definition of a call stack? Because software developers often use the stack as an analogy to the functioning of the human mind. You start one task and realize that to complete it, you need to do some other thing, but to complete that other thing, you need to first do a third thing, and so on. When this happens, your mind is acting like a call stack. What you really want to do is task A, but you are actually working on task C, because task C is needed to complete task B, which is in turn needed to complete task A. You have to keep track of all of this.
Let’s consider filling out your income taxes by way of an example. You want to complete your taxes on time, but to do that you need to have a list of all your deductions, and to do that you need to know what your out of pocket medical expenses were, and to do that you need a receipt that you are missing from your dentist, and to get a copy of the receipt you need to call them, and to call them you need to find their phone number. So you find yourself Googling local dentists rather than completing your taxes (frankly, you’d be better off texting them). In a sense you are completing your taxes by Googling for dentists, but there are some important risks you start to face when your call stack gets this deep.
One risk is analogous to stack overflow. Your stack gets so deep that you can no longer keep track of everything. It could be that you find yourself so far removed from your original task that you can’t remember what all the intermediate steps were, or you just abandon the whole affair because the original task starts to feel hopelessly remote. Here’s an example of what this can feel like, “Let’s see, I was originally sitting down here to work on my taxes, and I just now found that email from Frank from back in September, once I Googled a solution to getting my email search working again, and tried the third solution after the first two didn’t work, and I down-voted those and up-voted the third one, so I now see that the Dallas meeting was in September, and that means that my hunch was right, Jeff couldn’t have been there and hmmm… why do I care if Jeff was at the Dallas meeting? What’s that got to do with my taxes?”
Another type of risk of a deep stack is that one of these sub-tasks becomes huge and time consuming to complete. This is very common in software development. So common in fact that software developers have special names for this: a rat’s nest, a rabbit hole, or yak shaving. Whole companies have been built as a side effect of this common phenomenon. Developers come up with a good solution for problem C, so they can get task B done, so they can build product A, only to realize that the benefits of a solution to problem C actually make for a more compelling product (or at least one that’s already built).
This morning, I found myself in this ridiculous call stack:
13. Googling lots of links to put in this blog post because I’m a compulsive hyperlinker.
12. Writing this blog post because I realized how deep my call stack had gotten this morning, and feeling like it was a common and vexing problem that all of us, but especially developers, regularly face. Maybe by examining the problem, describing it, and thinking about possible solutions, I can help myself and others.
11, Googling for solutions on how to get Cabal (a Haskell package manager) to use gcc on the Mac when it’s not in /Developer/usr/bin/gcc. Answer: use a sym-link as specified here.
10. Using Cabal to install pandoc, a markdown converter written in Haskell.
9. Using pandoc to convert a textile gist I found and forked that documents Sublime Text 2 keyboard shortcuts
9 (aborted). Trying to get Marked to use RedCloth to render textile.
8. Trying to get Marked to render the local version of the shortcut reference.
7. Trying to move the shortcut reference, so I can edit and preview it locally in Marked rather than as a gist because I’m worried I’ll lose track of the gist.
6. Modifying the shortcut reference I found to contain just the shortcuts I’m likely to use in an order that makes more sense for me.
5. Googling for shortcut references for Sublime Text 2.
4. Looking up Sublime Text 2 keyboard shortcuts so I can get more comfortable with Sublime Text 2.
3. Working to get more comfortable with Sublime Text 2 so I can use it rather than TextMate.
2. Switching away from TextMate because the Clojure and Common Lisp support is not really up to snuff.
1. Setting up a reasonably productive Clojure development environment that’s not Emacs (my wife won’t let me grow a neckbeard).
Meta-goal: Start to work more with Clojure for use on some projects (and tangentially Common Lisp because I’m not a big fan of the JVM).
At some point I got deep enough in this stack to realize I was in trouble, but it took the reconstruction of this call stack just now to realize how much trouble I was truly in (I’d have guessed my stack was about 4 deep).
Q: Given a stack this deep, how much time has gone by?
A: About 3 hours.
Q: How much have I got accomplished towards my original task #1?
A: Almost nothing.
Q: How often has this happened with the task #1?
A: This is the second day in a row; different call stack, same result.
Q: Sitting there at task #11, what are my chances of getting back on task with the original goal?
A: Very low.
Q: How many of these calls where unavoidably necessary and how many of them could have been skipped to stay on task?
A: This is a judgement call, but from #6 on, I was pretty far off the rails.
Q: If any of these tasks turned into a rat’s nest and started taking more time than their value towards the original task warranted, would I have been aware and known to turn back?
A: No, in fact #6 turned into a rat’s nest but I plunged ahead anyway.
So what could have improved this? I think the solution to this problem lies mostly in increased awareness. If someone had tapped me on the shoulder, described the call stack I was in, and asked, “Is this a rational plan to achieve #1?” I would have blushed, said “no”, and immediately adjusted my activities.
I sometimes serve this role with my co-workers when I notice some task is taking a long time or when they come to me asking for help on some issue that seems very remote from what they are ostensibly working on. My first response is often to ask them to recreate the stack so we can decide together if some of it needs rolled back.
Usually no one is there to prompt us to be more efficient so we need to be able to prompt ourselves. We need to develop a self-awareness of this common and productivity sapping pattern. How might we do this?
I’m going to try doing it with a textual call stack. As I work, I’m going to jot little notes about what I’m doing, why I’m doing it, and when I started.
- 1 - 5:17 AM - Setup a productive Clojure environment
2 - 5:17 AM - search the web and read about Clojure/REPL/editor integration
- 2 - 5:25 AM - Setup Sublime Text 2
3 - 5:25 AM - D/L Sublime Text 2
3 - 5:26 AM - Read about Sublime Text 2 extensions
3 - 5:31 AM - Install Sublime REPL extension
- 3 - 5:31 AM - Configure Sublime REPL extension for use with Clojure
- 4 - 5:32 AM - Google error with Sublime REPL and Clojure
If this method, once iterated upon, proves to be effective, I’ll consider creating something that helps with the process. I can imagine that help with automatically entering the time and depth, calculating the elapsed time, “popping” the stack, getting alerts when too much time is spent on deep tasks (too much time probably being relative to depth of the task) and help with task switching (freezing one stack so you can resume work on another). All of these could probably be very beneficial to this process, but at this point the minimally viable tool to try this out is a notebook or text editor window in a very prominent place.
Is this a good idea? Let’s discuss it on HackerNews. If do you try it out, let me know if it helps you and I’ll be sure to report back on my results too.
Update: I’m going to use Tree for the Mac (a left to right outliner) rather than a text file so that “calling” and “popping” are easier and so my call stack depth (how far in the weeds I am) is more visually apparent.