The gospel of dogfooding
I’ve been asked to give a talk at a startup about dogfooding. The event is scheduled for March — but I’m using the holidays and multiple hours sitting on planes as an excuse to write down some thoughts on the topic; maybe it’ll help me be more coherent when I stand in front of them.
I have no idea how this particular company deals with dogfooding but this subject has come up enough times with enough companies that I figured it made sense to write a blog post about the activity and why the most successful companies follow it as dogmatically as a religion.
Dogfooding, in case you don’t know, is the simple declaration that we, the employees of this here company, will use our own product or service as part of the the daily process to get our jobs done — and we will do so before we expose our customers to it. Note that I didn’t mention “testing” but rather “getting our jobs done” on a daily basis. There’s a big difference…
We’re not just going to pay it lip service by running it in a few minimal configurations on an isolated network. We’re going to use it to run our own business. If it’s not good enough for us, it’s not good enough for our customers. It’s that simple.
But dogfooding is so much more than that. It’s an implicit pact that we, as a team, believe in the product so much that we will bet the company on it. We won’t ship the product until it meets that quality bar. No excuses; no ands, ifs, or buts.
What dogfooding is not: Putting it on a few slides as a goal for next year while knowing, full well, that someone somewhere in the organization will find a reason to give up trying. “We tried our best but ran out of time” is not an acceptable statement, ever, in a dogfood culture. At the risk of repeating myself: if it’s not good enough to run our business, it’s not good enough to run our customers’ businesses either.
It’s such a simple idea — but I’ve found, again and again, that companies go out of their way to convince themselves that it’s an optional step and can be skipped!
Don’t get me wrong. Dogfooding is required but not sufficient. It’s just one step in the process of getting a product out the door with high quality. But it’s a crucial one.
“They’ve done studies, you know. Sixty percent of the time, it works every time.” — Paul Rudd. Anchorman: The Legend of Ron Burgundy
But, in and of itself, dogfooding is not sufficient to prove the product’s readiness. Sooner or later, the product becomes so complex and has so many knobs and features that the employees’ use of it can’t possibly offer a statistically valid representation of the broader customer base.
When you have a million customers and a thousand employees, you can only get so far with them as your guinea pigs. There will be more bugs found by the beta community and the customer base. But that’s not an excuse to skip dogfooding as a necessary step in the quality process.
Dogfooding just says: Don’t you dare inflict this software, this service, this product onto your customers until you have forced your own employees to use it to get their work done, until you’ve bet your own company, your own employees’ success, on its quality. Put your money where your mouth is. Eat your own dog food. I dare you.
Here’s what you have to understand. Dogfooding is a religion, it’s a belief system.. Either you — and by extension, your team, your company — believe in that religion as one of the tenets of your survival, or you don’t.
“Elwood: You see, we’re on a mission from God.” — Dan Aykroyd. The Blues Brothers
If you believe in dogfooding, I bet you will do better in the long run: as an engineering team, as a product team, as a company. And if you don’t believe in it, please let me know now so I can short your stock.
Dogfooding is a belief system that your entire company has to buy into or you’re doomed to fail. Sooner or later. Most likely, sooner.
At its simplest, dogfooding is just the idea that you can use your own employees (and not just the engineers) as a test team. In this day and age where everything is implemented as an HTTP request and a cloud based service, some would argue that A/B testing delivers the same value so we can get rid of dogfooding. Bullshit.
Those A’s and those B’s are your customers. You are using them as your extended test team. Dogfooding comes before that. Consider the employees as your very first “A” group and we’ll have something to talk about.
As an engineering leader, it’s my job to figure out how to get employees to use our product. But I can only succeed if my peers believe in the necessity of dogfooding as much as I do, and my managers and CEO agree it’s a required step. They need to accept that it may impact release schedules, cause collateral damage in other teams, and even impact revenue.
I will only succeed if the employees, the guinea pigs, see the value as well. They need to buy into the experiment and help file bugs instead of complaining or finding ways to bypass the system. Anything less than complete buy-in from the whole company will doom the attempt.
Do you see now why it’s a religion, a belief system?
I learned this lesson at Microsoft — a company long vilified for its quality problems. The period I’m describing here was roughly 1998 to 2003 — the years when we, in the Windows team, were working on Windows 2000, Windows XP, and Windows Server 2003. At the same time, the Exchange team was working on Exchange Server 2003, the company’s enterprise class messaging solution — the server that talks to Outlook client software so you can send and receive email and schedule meetings.
The leadership team in Windows and Exchange had decided that the only way to improve the quality of both products was to force a hard dependency between them. The Windows team would henceforth be required to do all of its email and scheduling on pre-beta versions of Exchange Server 2003.
They had bought, whole stock and barrel, into dogfooding as a religion. There’s no question in my mind that that single step massively improved the quality of both products. It was hell to live through — but I applaud the execs for holding the line on quality.
I remember days and weeks spent with engineers and managers hanging around the hallways, bemoaning the fact that they couldn’t send each other email. I remember dozens of people showing up for meetings only to be told that it had been cancelled or rescheduled and a bug caused the cancellation to not show up on their systems.
We couldn’t send documents around for review because, at the time, email was our most common method of exchanging comments. When the Exchange servers crashed, we didn’t get to reboot them quickly. We had to wait for the Exchange team to debug the problem. Frustration was the word of the day.
But we were all in it together. Everyone agreed that improving quality was non-negotiable; if the software wasn’t ready for us, it wasn’t ready for our customers.
Some people took to hanging signs on their office walls: “No email for xxx days.” Delays upon delays, project schedule impacts, milestones missed, potential revenue impact. But, to their credit, the executives didn’t back off and give up on the idea of dogfooding.
Instead, they doubled down. We’re in this together. We’re helping our brethren in the Exchange team improve their quality. In fact, we’re going to take yet another cross-group dependency, requiring Exchange Server to run on beta versions of Windows Server. One good turn deserves another. If the Windows team can help improve the quality of Exchange, so too can the Exchange team help improve the quality of the next release of Windows.
Now that takes guts — especially at the scale of billion dollar businesses that we are talking about. By then, the combinatorics of features in Exchange and Windows, and the list of their cross-dependencies was so huge that there was no way we could test every combination.
Yet another reason for the eventual demise of shrink wrap on-prem software.
And it wasn’t just about Windows and Exchange. We also had to get other teams and third party vendors to support these pre-release versions of software: backup software, storage solutions, security software, clustering software, third party file systems and device drivers, you name it.
Some of these third parties were competitors and literally laughed us out of the room when we asked for their help. You want me to pull engineers off my own projects to help debug issues with some new version of your software that won’t even ship for another six months? Yeah, right. Pull the other one.
And yet, the executives didn’t back off. They had religion. They believed — there’s that word again — in their heart of hearts, that the only way they could attack the quality problems head on was to force us to eat our own dogfood.
And it worked. Out of that struggle came rock solid Windows and Exchange releases that I’m proud to have participated in.
The same model was used later for Office and SQL Server and Windows Client and other products, over time requiring all Microsoft employees around the world (all hundred thousand of them) to eat our own dogfood, to run our own pre-release software for releases such as Windows 7 before we would ship it in any form to our customers.
For what it’s worth, the reasons behind Vista’s failure were much more complex and I’ve already blogged about them.
That, in a nutshell, is dogfooding. You either believe in it, as a company, or you don’t. There is no way for a single team to be successful in the implementation of dogfooding if the rest of the company, all the way up and down the chain of command, doesn’t go along with that belief system.
The earlier in the lifecycle of your company (and product) you buy into this religion, the better. Adding it as an afterthought is much more painful. And anything less than wholesale belief in it as a requirement will usually result in its eventual failure in a company.
It’s so easy to find a reason why we just can’t take the time or make the effort to do dogfooding properly. I’ve seen far too many companies convince themselves that they don’t need to eat their own dogfood.
But we’d have to upgrade all the routers in all the buildings. But that version of the OS is not compatible with that version of the database with these patches installed. But we’d have to reprogram the firewalls and we just lost the only IT guy who knows how to do that. But we’re not like our customers. But we can’t afford the hit to revenue next quarter.
The list goes on an on. Sooner or later, the organization convinces itself that dogfooding is too hard, too much of a burden on the business, too disruptive to that other team over there, so let’s skip it. Just for this one release. We’ll see how it goes.
And there’s the rub — the entire belief system of dogfooding becomes optional. Within a couple of years, half the employees don’t even know what the term means and quality goes out the window as a result.
One last warning before you run off and convert to the religion of dogfooding. Attempting it without sufficient telemetry built into the system to guide your decisions is an exercise in futility. I mention this only because I’ve seen it derail far too many attempts but it’s a topic for a whole other blog.
“The average pencil is seven inches long, with just a half-inch eraser — in case you thought optimism was dead.” — Robert Brault.
Here’s another anecdote for you. I keep hearing about how Tesla is late with the Model 3, how customers who ordered early on still haven’t received their vehicles. I was surprised to hear all my friends complaining about this because I see dozens of Model 3’s on the road every day.
You see, I live within a few blocks of the World headquarters of Tesla. My only explanation is that the Model 3’s I see every day are driven by Tesla employees dogfooding their own product.
I applaud Musk for this. It’s not good enough for you until it’s good enough for our own employees. We should stop complaining about the delays and be thankful for the higher quality vehicles we’re sure to receive as a result.
Dogfooding is a required — but not sufficient — step in the delivery of high quality software and services. In order to be successful, it needs religious buy-in to the whole belief system from everyone in the company.
Can I hear an Amen, brother?
