Shaving the Yak (or why software development is so distracting)

tldr; software development is prone to distractions as new tasks spin out from the original task. Developers should try a productivity aid (just a notebook, text editor, or outliner to start) to help them manage this phenomenon, mainly by increasing their awareness of the issue as it happens.

For non-developers that might be reading this, the call stack (or just stack) is the place where an executing program keeps track of what it is doing. When part of a program calls to another part of the program, an entry is placed on the stack so that when the other part of the program is finished, the program returns back to the original point where the call was made, and the program continues on from there. This layering of the stack as one procedure calls another which calls another, and so on, can get very deep. The stack can get so deep in fact, that the execution of the program must be abandoned. When this happens it is called a stack overflow (which is where the name for the software development Q&A site comes from).

Why mention this (simplistic) definition of a call stack? Because software developers often use the stack as an analogy to the functioning of the human mind. You start one task and realize that to complete it, you need to do some other thing, but to complete that other thing, you need to first do a third thing, and so on. When this happens, your mind is acting like a call stack. What you really want to do is task A, but you are actually working on task C, because task C is needed to complete task B, which is in turn needed to complete task A. You have to keep track of all of this.

Let’s consider filling out your income taxes by way of an example. You want to complete your taxes on time, but to do that you need to have a list of all your deductions, and to do that you need to know what your out of pocket medical expenses were, and to do that you need a receipt that you are missing from your dentist, and to get a copy of the receipt you need to call them, and to call them you need to find their phone number. So you find yourself Googling local dentists rather than completing your taxes (frankly, you’d be better off texting them). In a sense you are completing your taxes by Googling for dentists, but there are some important risks you start to face when your call stack gets this deep.

One risk is analogous to stack overflow. Your stack gets so deep that you can no longer keep track of everything. It could be that you find yourself so far removed from your original task that you can’t remember what all the intermediate steps were, or you just abandon the whole affair because the original task starts to feel hopelessly remote. Here’s an example of what this can feel like, “Let’s see, I was originally sitting down here to work on my taxes, and I just now found that email from Frank from back in September, once I Googled a solution to getting my email search working again, and tried the third solution after the first two didn’t work, and I down-voted those and up-voted the third one, so I now see that the Dallas meeting was in September, and that means that my hunch was right, Jeff couldn’t have been there and hmmm… why do I care if Jeff was at the Dallas meeting? What’s that got to do with my taxes?”

Another type of risk of a deep stack is that one of these sub-tasks becomes huge and time consuming to complete. This is very common in software development. So common in fact that software developers have special names for this: a rat’s nest, a rabbit hole, or yak shaving. Whole companies have been built as a side effect of this common phenomenon. Developers come up with a good solution for problem C, so they can get task B done, so they can build product A, only to realize that the benefits of a solution to problem C actually make for a more compelling product (or at least one that’s already built).

This morning, I found myself in this ridiculous call stack:

13. Googling lots of links to put in this blog post because I’m a compulsive hyperlinker.

12. Writing this blog post because I realized how deep my call stack had gotten this morning, and feeling like it was a common and vexing problem that all of us, but especially developers, regularly face. Maybe by examining the problem, describing it, and thinking about possible solutions, I can help myself and others.

11, Googling for solutions on how to get Cabal (a Haskell package manager) to use gcc on the Mac when it’s not in /Developer/usr/bin/gcc. Answer: use a sym-link as specified here.

10. Using Cabal to install pandoc, a markdown converter written in Haskell.

9. Using pandoc to convert a textile gist I found and forked that documents Sublime Text 2 keyboard shortcuts 

9 (aborted). Trying to get Marked to use RedCloth to render textile.

8. Trying to get Marked to render the local version of the shortcut reference.

7. Trying to move the shortcut reference, so I can edit and preview it locally in Marked rather than as a gist because I’m worried I’ll lose track of the gist.

6. Modifying the shortcut reference I found to contain just the shortcuts I’m likely to use in an order that makes more sense for me.

5. Googling for shortcut references for Sublime Text 2.

4. Looking up Sublime Text 2 keyboard shortcuts so I can get more comfortable with Sublime Text 2.

3. Working to get more comfortable with Sublime Text 2 so I can use it rather than TextMate.

2. Switching away from TextMate because the Clojure and Common Lisp support is not really up to snuff.

1. Setting up a reasonably productive Clojure development environment that’s not Emacs (my wife won’t let me grow a neckbeard).

Meta-goal: Start to work more with Clojure for use on some projects (and tangentially Common Lisp because I’m not a big fan of the JVM).

At some point I got deep enough in this stack to realize I was in trouble, but it took the reconstruction of this call stack just now to realize how much trouble I was truly in (I’d have guessed my stack was about 4 deep).

Q: Given a stack this deep, how much time has gone by?
A: About 3 hours.

Q: How much have I got accomplished towards my original task #1?
A: Almost nothing.

Q: How often has this happened with the task #1?
A: This is the second day in a row; different call stack, same result.

So what could have improved this? I think the solution to this problem lies mostly in increased awareness. If someone had tapped me on the shoulder, described the call stack I was in, and asked, “Is this a rational plan to achieve #1?” I would have blushed, said “no”, and immediately adjusted my activities.

I sometimes serve this role with my co-workers when I notice some task is taking a long time or when they come to me asking for help on some issue that seems very remote from what they are ostensibly working on. My first response is often to ask them to recreate the stack so we can decide together if some of it needs rolled back.

Usually no one is there to prompt us to be more efficient so we need to be able to prompt ourselves. We need to develop a self-awareness of this common and productivity sapping pattern. How might we do this?

I’m going to try doing it with a textual call stack. As I work, I’m going to jot little notes about what I’m doing, why I’m doing it, and when I started.

• 1 - 5:17 AM - Setup a productive Clojure environment
• 2 - 5:17 AM - search the web and read about Clojure/REPL/editor integration
• 2 - 5:25 AM - Setup Sublime Text 2
• 3 - 5:25 AM - D/L Sublime Text 2
• 3 - 5:26 AM - Read about Sublime Text 2 extensions
• 3 - 5:31 AM - Install Sublime REPL extension
• 3 - 5:31 AM - Configure Sublime REPL extension for use with Clojure
• 4 - 5:32 AM - Google error with Sublime REPL and Clojure

If this method, once iterated upon, proves to be effective, I’ll consider creating something that helps with the process. I can imagine that help with automatically entering the time and depth, calculating the elapsed time, “popping” the stack, getting alerts when too much time is spent on deep tasks (too much time probably being relative to depth of the task) and help with task switching (freezing one stack so you can resume work on another). All of these could probably be very beneficial to this process, but at this point the minimally viable tool to try this out is a notebook or text editor window in a very prominent place.

Is this a good idea? Let’s discuss it on HackerNews. If do you try it out, let me know if it helps you and I’ll be sure to report back on my results too.

Update: I’m going to use Tree for the Mac (a left to right outliner) rather than a text file so that “calling” and “popping” are easier and so my call stack depth (how far in the weeds I am) is more visually apparent.

development

productivity

Snooty Monkey: Andrew’s Perspective

Guest post by Andrew, one of the Snooty Monkeys.

I met Sean the same year that Snooty Monkey was started. At the time, I was working on a digital textbook application called ezLearnz, while working at a day job that had nothing to do with software. Sean was also working on an education app called Live Syllabus, while working a day job at IBM. We saw the possibility of a 1 + 1 = 3 situation… known as a synergistic relationship in the corporate world we were working in at the time.

Sean and I met regularly on Saturday mornings for more than a year. During this time, Sean mentored my process of becoming a software developer and we discussed educational topics, including an idea very similar to what is now P2PU. We realized we were both passionate about products that simplify and facilitate education and learning. In the end, we did not continue to pursue either of our education apps; we realized the education landscape was changing too rapidly, and what was missing from existing products was a revolutionary user interface that would radically improve users’ learning experience. During this time, I started getting interested in self-improvement apart from our discussions on the subject.

A year later, I started contracting for SnootyMonkey. Our discussions on education continued, but at this point broadened to self-improvement, philosophy, maximizing global happiness, and other interesting thoughts.

Recently I’ve become interested in the softer skills of web app development, particularly marketing. I like the idea of becoming a well-rounded web app generalist… being able to not only create an awesome app, but find people willing to pay for it. Luckily, Snooty Monkey happens to be in need of a suitably Snooty (yet open, transparent, and ethical) Marketer. And applying Snooty Monkey’s new tagline, “Simple Self-Improvement for a Complex World” to my own career, while improving Snooty Monkey, is just so deliciously recursive. Stay tuned for more Simian Snootiness as I work to figure out ethical web marketing!

A New Direction for Monkey Opus

The Snooty Monkey tagline is a common household phrase of course. But just in case anyone reading this is a recent immigrant from Papa New Guinea, or just concluded a multi-year NASA space travel deprivation experiment, the tagline is, “Simple Software for a Complex World”.

As of this moment, Snooty Monkey has a new tagline, “Simple Self-Improvement for a Complex World”. Why the new tag line? The old tagline reflected a clear mission in 2008, let me take my skills as a “software product person” and build a self-supporting consultancy, lest my family starve to death. The new tagline reflects an aspirational change here at the end of 2011, and it better fits the broader original goal of Snooty Monkey, which was to create a bootstrapped software company.

This brings me to the actual topic of this blog post, changes in Monkey Opus (which happens to be the very blog you are reading). In the past Monkey Opus has not been terribly focused. There has been a mix of entrepreneurship, product management, and technology topics. Moving forward, the focus will narrow. It’s not that entrepreneurship, product management, and technology topics will never be discussed here. I certainly won’t be able to always resist that temptation, but the major focus of Monkey Opus is now self-improvement.

Why self-improvement? Because it’s what Snooty Monkey is all about as a software company. We are all monkeys. This life of ours is our opus. Let’s be snooty about it and make it a good one!

Snooty Monkey: Lessons Learned

Snooty Monkey was started in 2008 as an experiment. The focus of the experiment? Can a CTO / Lead Developer / Product Manager type (that’s me by the way) create a hybrid micro-ISV / consultancy company? One that purposely stays small (just me? me and 2?), while serving a niche client base (bootstrapped entrepreneurs) with a niche service (temporary technical co-founder). Can doing this afford me the time to experiment with software products of my own without the traditional startup pressure that each one be a hit or the whole company fails?

Unlike a good experiment, where things are tested one at a time and variables are controlled, by necessity this experiment had lots of variables and was chaotic at times. The grand experiment is far from over, in many ways it’s just begun, but some of these experiments have now run their course, their questions have been answered and the results are ready for review and confirmation by the rest of the scientific community. It’s time to refocus the lab on new questions.

So, what has been tested so far with Snooty Monkey? What were the hypotheses? And what can now be concluded?

• Experiment: Can I support my family without a “job”?
• Hypothesis: Yes.
• Conclusion: Yes. I’ve worked for myself since early 2009. I have yet to take the plunge and hire an employee, but Snooty Monkey is the main source of income for a few contractors (I call them friends though).

• Experiment: Is there a market for a freelance “temporary: technical co-founder / CTO / lead developer / product manager” in the bootstrapped entrepreneur community?
• Hypothesis: Yes.
• Conclusion: I can say in no uncertain terms that many non-technical founders of bootstrapped startups will be willing to partner for some or all of the technical aspects of their business so they can focus on what they do best, eventually replacing Snooty Monkey with their own technical team when the business finds product/market fit and needs to scale. If you have the skills, drive, maturity and experience to make a good technical co-founder, there is a market for you services for cash rather than equity. In these 3 years, about 80% of the work and about 90% of the revenue for Snooty Monkey has more or less fit the experimental description. At times I’ve done work for big companies (mostly sub-contracted through interactive marketing firms) and for venture-funded startups, and almost without fail, this work has been less rewarding and fulfilling for me and less successful for the company. At the end of the day, I’m a technical entrepreneur. If I’m doing something else for you, you’re probably not getting as much bang for your buck as you should and I’m probably not having as much fun as I should.

• Experiment: Can I pick and chose the projects I want to work with to fit my technical interests?
• Hypothesis: Yes.
• Conclusion: Yes. When I left the Biggest Blue Java shop, I made a promise to myself not to work with technology I don’t like. I’ve built many Ruby on Rails and Objective-C products, and I’ve got projects with Erlang, CoffeeScript and even Prolog in them, all of which I really enjoy. The couple of side trips into C++, Flash, Flex, and Java have been few, brief and mostly self-inflicted (Ouch, I forgot how much that sucks! Note to future self…).

• Experiment: Can I pick and chose the projects I want to work with to fit my interests?
• Hypothesis: Yes.
• Conclusion: Mostly no. My main interests are self-improvement, philosophy, literature, sports, alternative software development technology, and education. I’ve had some luck building products in the field of sports, but other than that, I’ve not had much interest aligning my interests with my bootstrapped entrepreneurs’ product needs. I’ve built many branded marketing apps. I’ve built scientific apps, quality assurance apps, customer service apps, healthcare apps, etc. All of these have been very interesting in their own right and I’ve learned a ton with each one. But when I work on my own products as Snooty Monkey, I have a perfect fit with my own interests (Snooty Monkey’s two current products are in self-improvement and sports). This kind of alignment is a powerful force that should not be undervalued.

• Experiment: Can I mantain my own high ethical standards while contracting my time to others?
• Hypothesis: Yes.
• Conclusion: Mostly yes. I’ve been extremely impressed with the character of the entrepreneurs I’ve been fortunate to work with. I regularly hear and read about startup horror stories starring villainous and greedy entrepreneurs. But the entrepreneurs I’ve met and agreed to work with have been some of the most upstanding people I know. They’ve been impeccably fair to me and to their customers. I’m close to suggesting a hypothesis that entrepreneurs are better people on average than society at large would lead you to expect. The one exception to charting my own ethical course has been in the area of software patents. Unfortunately I’ve not seen eye to eye with some of my very best clients about the nature, value and ethical standing of the broken US patent system. The work I’ve done in just the last 3 years has led to over a dozen patent applications, many of which I directly helped formulate (Snooty Monkey does work for hire so those patents belong to the clients). I’m not proud of this. It is a case of my deeds not living up to my thoughts and words. I’m considering declining to directly participate in software patent activities in the future. This could cost Snooty Monkey valuable business.

In looking back over these questions, I find it a bit difficult to remember how scary and unknown they seemed at the time, I can’t help but be pleased at how things have turned out.

The risk I now face is that I become complacent. I’m my own boss, I have a decent income, and I get to work with great people, my team of contractors and bootstrapped entrepreneurs, who want nothing more than for me to succeed. After all, me succeeding is the only way my clients get to succeed. There are no politics, no ulterior motives, and no bureaucracy. It’s the business of software in its purest form. Just them and their idea and me and my keyboard.

In some sense then, I’ve become too successful at what I do now, which is helping others build software companies without taking or before taking investment. This is now an 80-hour-a-week job for me. I could do nothing else for the foreseeable future. But it’s not what I set out to do. It was all a means to an end. That end being to build my own bootstrapped software company. And not just any software company. I want to create software products that help me improve my life, and if all goes well, help others improve their lives in the same way.

I’m ready to admit that I’m no longer making any progress on this. I’ve done some really amazing and cool things since starting Snooty Monkey. I wrote a book, I created some great products, I’ve even helped create what are now some great companies. But I need to get this train back on the original track. I need to better balance the two parts of my business. I need to manage my goals, my time, my tasks and my attention to achieve more. I need to improve myself. I need to be better.

I owe this to myself, I owe it to my family, I owe it to Snooty Monkey’s contractors, clients and current and future customers.

During the last couple of weeks I’ve been working with Andrew (one of said contractors) on the seeds of a new plan. This plan is going to germinate and change and grow (or shrivel and die) here in the open on this blog. Get your popcorn ready! It’s going to be an exciting show.

Build Erlang R14Bx on Mac OS X Lion

Usually to build software from source on Linux or Mac OS X, you follow the pattern:

./configure
make
sudo make install

When trying this on Mac OS X Lion you’ll find that make is no longer part of the OS like it was with Snow Leopard and before. Instead, it is part of Xcode 4.1 or greater. So you need to head to the Mac App Store app and get your free copy of Xcode 4.1 (or greater) so that you’ll have make installed.

Another wrinkle, specifically for getting Erlang from source, is that there are some obscure flags you need to use with configure to get Erlang to build cleanly. This sequence works (tested on Erlang R14B03 and Lion 10.7 (11A511):

CFLAGS=-O0 ./configure --enable-darwin-64bit
make
sudo make install

Thanks to the community on the Erlang questions mailing list for collaborating on this solution. I’m just passing it along.

Erlang

mac os x

Lion

Bringing Serendipity Back

If Timberlake can bring sexy back, then I guess dictionaries can bring serendipity back too. I’m a regular user of online dictionaries and they all seem about the same to me (which is to say, somewhere between pretty bad and awful) so I use them through Google rather than having a go to favorite. I search for the word and then pick one or more of the top 10 hits to look at.

Today a search on the word “monotonist” (I know what it should mean, but I said it the other day, and then wondered if it was actually a word or if I just made it up) brought me to this fairly gross site - http://www.websters-dictionary-online.com/definition/monotonist (linked with nofollow to avoid giving them any link juice). I’m certain, due to the suspicious lack of branding, the poor performance of the page loads, and the dubiousness and quantity of its ad content, that it’s not actually associated with Webster’s (which is here - http://www.merriam-webster.com/).

What caught my eye was this part of the page:

How cool! I can’t begin to count how many hours I spent unintentionally reading the dictionary as a kid. I’d ask my parents how to spell a word, or what a word meant, and they’ d inevitably answer “d-i-c-t-i-o-n-a-r-y” or “it means you should look it up”. I’d traipse over to the family set of the Funk & Wagnall’s encyclopedia (bought from the grocery store on subscription, those were the days!) and its accompanying dictionary (in 2 volumes) and look up the word. Once I found my word, I could never resist the siren call of the other words on the page. Mixed among the handful of words I knew were amazingly short words, gigantic polysyllabic monstrosities, foreign words without enough vowels or with too many vowels, and words with accompanying black and white pictures, line drawings or maps. Each trip to the dictionary was a multi-paged odyssey through the English language.

Sadly, I doubt my kids, lifelong online dictionary users, have ever experienced this. The Web was invented to provide better context; the raison d’etre of the hyperlink and the subsequent Web server and Web page was to provide context to referenced concepts with just one simple click. But what these online Web dictionaries don’t have is page context. What else is on this dictionary “page”? What is on the “opposite page”? The “next page”? The “previous page”? And why are these words near one another? Is there a shared prefix or root? A shared word of compound words?

Links and the power of the Web to provide context are used in every online dictionary page and it would be hard to argue that the Web doesn’t provide a much better experience for exploring a specific word. You have links to antonyms and synonyms and etymology and stems and related words, and, and, and…  But what about the words that are “physically” near this one? I haven’t seen an online dictionary attempt to provide this until stumbling on this shady site.

The “nearness” of words is an important concept in my brain’s structure and in my understanding of English. I don’t think I’m alone in this, but I do think that in the span of just one generation we’ve probably lost that part of the organizational structure of our language in our brains. It’s a shame. We shouldn’t go back to paper dictionaries of course, but there is something for the other dictionary sites to learn from and improve on in this site’s listing of adjacent words.

Apple is Not a Software Company

Marco Arment, the founder of Instapaper and former tech lead at Tumblr recently wrote that there really isn’t much of a market for tablets, just for iPads. I have no bone to pick with his main point, he’s right about that. But part of his analysis is way off base (the bold emphasis is mine):

And, like they’ve done so far with Android, the hardware manufacturers will continue to attempt to make their own user-facing applications and front-end interfaces, but these usually suck. (Other people may describe “suck” with kinder, apologist adjectives like “getting better” and “not bad”.)

These manufacturers aren’t software companies: they’re hardware companies that write software out of necessity. Apple is a software company that makes hardware out of necessity. The software side of a modern computing platform is far more difficult and expensive to create and maintain than the hardware. Anyone can cobble together the same processors, DRAM, flash, and radios as Apple, put them into a plastic case, and run a commodity OS on them with slight front-end customizations. But not everyone can create an entire software platform.

Apple is absolutely not a software company that makes hardware out of necessity. That sells Jonathan Ives and his team of brilliant industrial engineers short. Creating innovative hardware is still very hard. Copying Apple’s hardware is easier, but still difficult; it takes a few tries to get right.

I can see where Marco is going with this, he wants to paint a simple picture of two types of companies, hardware and software companies. He wants to put Apple, Google and Microsoft into the software category and all the other tablet makers that use Windows and Android such as HP, Dell and the Asian manufacturers into the hardware category. He then wants to say good software is really hard and good hardware is really easy, so the tablet manufacturers that aren’t software companies are stuck. Apple makes their own hardware so it works fantastically well with their software, but the hardware companies are forever disadvantaged because they can’t make software that will will work fantastically well with their hardware.

This is just not an accurate picture though, Apple is not a hardware company dabbling in software or a software company dabbling in hardware. Apple is the last of a breed of something we used to have a lot of, a systems company. Systems companies create systems composed of hardware and software.

I’m not sure how old Marco is, but I’m guessing he’s a bit younger than me because I can remember a time when the computing landscape was dominated by systems companies: IBM (z/OS, RS/6000 and AIX, PS/2 and OS/2), DEC (VMS), HP (HP 3000, HP-UX), SUN (SPARC and SunOS/Solaris), Silicon Graphics (IRIX), Apple (Mac), Be (BeOS and Be Machine).

In a classic disruptive move, Microsoft and it’s flock of PC OEM’s (Dell, Gateway, Acer, etc.) killed off the systems companies one by one, either outright, or by forcing them to abandon their systems (HP, IBM except for the mainframe) in favor of becoming a Windows OEM. Apple is the only one that survived as a systems company at a time that didn’t favor the systems companies; hardware compatibility was king and hardware innovation was static (More’s Law was driving all the innovation).

Now that Moore’s Law is breaking down and innovation in hardware form factors and interfaces is back on the top of everyone’s agenda, it’s a great time to be a systems company again. Apple is outperforming their technology peers, and we all want iPad systems, not some hardware devices running Android or Windows software.

Apple