The Limits Of A Data-Driven Approach

Because we lack the conventional metrics to define and measure, for example, the hardships of walking, we don’t design and enforce solutions or adopt targeted public policies.

Designers in almost all fields make critical decisions based on their domain knowledge, observable conditions, and of course data. More and more, incredible amounts of finely grained and immediately available (and updated) data is required to be incorporated into product decisions. Given the quantitative nature of some of this data, it is often judged to be unbiased, truthful, and complete. Few people take the time to question the data - not the accuracy per se, but what is (and is not) being measured.

This article highlights the assumptions and challenges urban planners encounter when considering the health of our cities. It also shows how qualitative data can be used to make clear problems that purely quantitative approaches often fail at capturing. Indeed, its hard to imagine the level of instrumentation required to provide the nuance captured in a stroll down a city street.


Designers, not Dribbblers

Paul Adams has written a fantastic short article (I suppose it might even be considered long-form these days, but that's another topic) about what he calls The Dribblisation of Design. In it, he derides the growing trend, as he identifies it, of designers posting attractive mockups and reworks of existing or fictitious products on Dribbble, a gallery of member's work, in the absence of any context. By context, he means the business environment that almost all substantial product work takes place in. He's not knocking the desire of designers to show off their work, but the undue influence that showing work out of context can have.

Dribbble itself shapes the conversation to some extent, the medium shaping the message, with highlighting of colour palettes and other superficial details prominent in the UI. People look and people emulate.

Designers can easily retreat into pushing pixels because "it’s just more fun to draw nice pictures and bury oneself in pixels than deal with complicated business decisions and people with different opinions."

Regrettably, this is a problem that has always existed and will never go away. Designers come from all sorts of backgrounds, with a huge array of skills and experiences informing their practice. Richard Buchanan, one of my professors at CMU talked about design practiced across four orders, and while he emphasized that these should not be considered as existing on a continuum or as outcomes, it is a useful framework for understanding the breadth and depth of what a contemporary product designer should be addressing:



Strategic Planning

Systemic Integration


Signs, symbols, and images





Physical Objects




Activities, services, and processes



Systems, environments, ideas and values

Most designers aren't trained to do more than decorate - to spend time in the upper left, where their skills are most easily displayed and recognized. The woeful lack of preparation designers are getting in most schools is a serious issue. I think Paul is addressing an elite group of product designers in this piece, not the vast majority of designer/decorators. Not mentioned in Paul's piece are the structural issues within most contemporary product development environments that favor the atomization of output, further exacerbating a designers worst habits. He does talk about the need to align work to company vision and provides some excellent guideance on how to frame that understanding. Processes that disallow for a holistic view of the business problems and user goals are only going to yield fragmented, disconnected experiences. They might look great, but they won't be solving the real big problems designers should be.

Design or Cut Bait

At a UX conference not so long ago, a fellow presenter (a non-designer) and friend remarked that in so many of the presentations, he detected a tones of exasperation, almost whining, coming from the speakers. He wondered why so many designers were concerned with not being in a strategic or decision-making position, when the opportunity is theirs for the taking. I've encountered this quite a bit, and confess to being that designer in the past. At a lot of companies, especially larger ones, the goals and language set are dominated by people with backgrounds in business, sales and marketing. Much of what is discussed has an aura of rationality, science or math, its largely subjective, and its difficult for many designers to overcome the wall of jargon they encounter. Designers definitely need to learn some of this to approach the table, but they should be prepared to bring their own. Look for ways to shift the culture, ways to open up the discussion to include stories from real users, and ways to build on what's already being done.

Take a hard look at where you are working - is it even possible to achieve this? In my experience, if there isn't someone a couple rungs above you who gets it, you are in for a long, frustrating and probably futile quest. If this isn't working, find another company that does get it, work in consulting, or start your own. Agency work is appealing to designers partly because someone else is paid to inject them into the position they covet, but also because someone is running interference for them. They get to focus on design work while someone deals with the challenging clients.

I vastly prefer working in-house, where I see a rich range of problems and get the chance to see solutions through to conclusion and revision. It's a personal preference, and one that requires patience and stamina in different ways than other contexts.

Agile UX NYC Slides

On Saturday, I had the pleasure of speaking at the inaugural Agile UX New York conference here in New York. The roster of speakers was excellent and I learned quite a bit. I spoke about some of what I've learned at betaworks over the last four years working in an early stage startup environment. I outlined a number of the elements I saw as contributing to the success of some of the companies I've worked with to get up and running. Please take a look at my slides below:

Bonus - If you can bear it, here is a video of me giving this talk, via Ustream.

Speaking at Agile UX New York City

I'm excited to announce that I'll be speaking at the upcoming Agile UX conference here in New York. I'll be talking about the how the relationship of design to the world of startups has recently shifted from a question of necessity to a position of criticality. To succeed in this new environment, designers need to adapt their strengths. Specifically, I'll talk about my experience at betaworks and how designers innovate in an early-stage startup environment and transform ideas into products then companies. Join me if you can!

Who "Gets" Product?

Like many in my field, I'm always amazed when poorly conceived or executed products find their way to market. While every case study of failure is unique, starting with a great product team is a variable we'd like to have under control. Finding people who work in product development with a compatible outlook and skillset is difficult, but identifying higher-order abilities in those people is hard. How do you know if someone “gets” product?* You want to find these people, but what are you really looking for? This is a deceptively hard question, and the easy (but unsatisfying) answer is that you can't. The other easy answer is that there are many answers. I've shared my own perspective, but I also asked a number of people to hear what they thought. What I Look For - The Short List

  • X-ray Eyes People I know that get product can "see through" a product along multiple dimensions to understand all of what goes into making it and where it can go. What the decisions were, the trade-offs and meetings during the process of development. How many times did they test a part, and did they fix it? What will happen over time? How are they planning for the unknown?
  • Mostly Makers Skills in making, editing, and curation are very important to me, but are only part of a holistic skill set and outlook (and many great product people aren’t makers). Curiosity about how and why things work and succeed (or fail - why does Hollywood make so many bad films?). A good track record helps, but being flexible about what success is may be necessary. Some of the best product people I know I’ve known for a long time, but its hard to get that insider perspective.
  • Well Spoken I like it when someone can articulate the stance a product takes. Is a company trying to break out or fit in? They see how people use it (can they use it, is it meaningful, do they like it, will they keep it) now and in the future, and everything orbits around that. More literally, can people talk about products with clarity and directness (and metaphor). Many fields have a specific language set so insiders can be very specific, and product people should be well-versed or be able to adopt the local language.

From the Experts I asked several friends and colleagues to share their experiences, and was delighted with their responses. Several commented on the difficulty of the question itself, but all took up the challenge. I’ve synthesized their responses below, but thanks to Charles Adler, John Borthwick, Dan Boyarski, Liz Danzico, Alex Rainert, and Khoi Vinh for taking time to respond. Here are their key points:

  • Its About People People that get product understand that fundamentally this is about people. Product people use products. They talk about products in the context of use (as opposed to the features) and about the emotional engagement that exists for them. Development is a human process, and requires an understanding of the interaction of the roles involved and, of course, who the audience is.
  • It Takes Holistic Thinking Getting product also requires (or may be an outcome of) holistic thinking. They think about all aspects of the product: market, technology, operations, support, design. They can talk about and balance the relationships among them.
  • Bring a POV Despite being able to balance across disciplines and requirements, they have opinions that they hold strongly and can trust and defend them. They can say smart things about products - their own and other people’s. They understand where they’ve failed and can build on that.
  • Prove It Being able to demonstrate the ways they go about solving problems is important. Seeing past work is one measure, and seeing the results of in-person problem solving is used often. They understand the roles required, and they actually have experience shipping something.
  • Legacy Perhaps the most elusive, but in some ways critical quality, is whether someone can be trusted in the future to continue, extend and grow a product.


*By “getting product”, I mean people who can understand how and why products are made and succeed (or don’t), and can articulate and repeat that outcome.

Whole-team User Research

In the 15 or so years I've been designing interactive products, I have always tried to involve the intended audience (users) in the course of product development. A recent article by Jared Spool on articulates something different that I've felt but rarely had the data to support: involve the rest of the team in deep exposure to users. I'm in complete agreement with this approach, as long as sensitivity to team structure and dynamics are understood and managed well. Resistance to conducting research (broadly speaking, from field research to usability testing) can come from unexpected quarters, but the usual suspects include Sales (who fear losing control of the relationship they have with their clients), Technology (who look at it as something that will slow down develpment time), and Product (who've invested too much time in their MRD and PRD documentation). The Design team, too, has their share of those who fear user input will impinge on their freedom to create, among other reasons.

I started my career at Morningstar working on Windows-based software, where our team used a waterfall process and testing was difficult. We had to wait until the product was nearly done, which was the only way it could stand up long enough for us to test it with users. Even then the feedback was invaluable, but we rarely had many of the front-line developers on hand to observe. Also, much of the development staff were from mainland China and had a more limited command of English.

Later on at Razorfish, I learned valuable lessons on how not to run user research. One of the failures of the agency model is how often work is done in an assembly-line fashion, with little room for iteration and teams that are separated from each other. My team did excellent work, but too often we weren't tightly coupled with the IA, Design, and Tech teams to provide user feedback in a timely manner. It was also trimmed or eliminated from pitch budgets since it always appeared as a line item.

At Yahoo!, we had a much more tightly integrated UX team, but we weren't physically or organizationally close to Tech. In between us was a firewall of Project Managers (necessary in any large endeavor, I might add). This inhibited our ability to get everyone on board, although we did manage to get some of the front-end developers to some usability tests.

Right now I'm at betaworks, and the situation is different still. Here teams are very small and fast-moving. I don't have a team and everyone is super-busy and its hard to find time to squeeze in anything other than core job responsibilities. There are vastly different outlooks on what roles are critical, and products serve very different markets (although they all have UIs). In a startup environment, there is a lot of pressure to maintain a clear vision and course in the face of all manner of obstacles, users included.

Here lies the challenge: how to integrate user feedback into a team's work so that its not considered unusual or can be sidelined. A few takeaways:

  • Convince teams that something that feels so counter-intuitive is actually better. Spending more time on understanding users, planning and designing actually makes the development cycle shorter (and saves tons of time post-launch).
  • Convince teams that this is as important a part of the process of developing interactive products as continually testing for and fixing bugs.
  • As Spool points out, every discipline needs to be involved to avoid the kind of infighting that comes about when people are basing decisions on assumptions about user goals, not empirical evidence.

Design decisions for

The chartbeat dashboard recently underwent its first major revision since launching a year ago. Below is a copy of a post I wrote for the chartbeat blog giving some background into the redesign. The team has a strong vision for chartbeat, and to bolster our vision I led some quick-and-clean research into how current and prospective users view chartbeat. Our plan included heuristic evaluation, in-person usability reviews, and group-based cognitive walkthroughs, all to provide insight and momentum. The team arrived at a set of first principles to guide development, and we quickly settled in an iterative design-build-test cycle, where the fidelity of each step evolved as we built out the site. I've highlighted some of the core principles and selected design decisions below:

Structure the site around user goals chartbeat is a tool for front-line workers, not just internal analytics teams. We designed the layout around a set of use cases, so users could walk through the data in a logical fashion, understand causality, and take action. The triad of panels at top allows users to understand "how many people, how did they get here, and what are they looking at?" in a snap. We also heard our users love the kinetic nature of chartbeat, so we extended that a bit with a Matrix-like stream of raw hits in the right-most column.

Data should be appropriately dense, clear and actionable Data should be rich and deep, without compromising ease of use and clarity. As an example, the tree map in v.1 was challenging for users - they liked the intent, but it was difficult to interpret. Sites with extremely low or high traffic or with few pages skewed the chart so that it was impossible to analyze. We decided to use a small range of fixed sizes, ensuring the display of most pages, and using dots to represent visitors. Larger numbers and page titles increase legibility, while isolating the page modules with white space makes it easier to read them as units. We also standardized and gave meaning to the range of colors we used, so users can more associate meaning across the panels.

Everything should be on a single page A key interaction design challenge chartbeat faces is letting users drill into richer data whithout resorting to traditional hierarchical navigation schemes. We came up with the notion of "pivoting" around a selected data element, where the entire page changes to reflect just that element. This way, chartbeat can serve as a site-level analysis tool and easily shift to isolate a page with a single click.

Use historical data as context, but keep it a real-time tool Users love the real-time aspect of chartbeat data, but a unanimous request is to provide some context to what they're viewing. Some amount of historical data was in the previous version, hidden behind a tab at the top of the page. Hidden, too, was a powerful replay feature that lets users isolate events and walk through it ("Tivo for your website" as Tony likes to say). By bringing the replay to the fore, we signaled to users that historical data is available by showing a trend chart. When users pivot on data, we also show a thin historical chart that can be expanded for more detail.

We've been doing some followup visits with users to understand their long-term usage and how it is fitting into workflows. One big finding has been that the clarity of the data brought many new features to the foreground for users, giving them new reasons to use chartbeat.

On the iPad

It's been a couple of days since Apple released their latest product, the iPad. Given the denouement, its not surprising that there is a fair amount of disappointment. I'm not talking about the name, but the feature set. I see it this way - when the fastest-growing areas of your company are your media outlets (iTunes and the App Store), you build on that success. While laptops have been doing quite well, overall the growth of the traditional computer business for Apple hasn't been as impressive. Apple Computer even became just Apple not so long ago.The iPad is a media consumption product first, powered of course, by a computer. There have been plenty of tablet computers that have failed, and I suspect that failure has much to do with the "open-ended"ness of the devices - they are basically laptops jammed into a different form factor. Apple aficionados seemed to be clamoring for a Mac Air without a keyboard - a computer in its unvarnished chameleon-like natural form. I'm actually with them, but I'm probably in a too-small (though vocal) market  for them to target. Its my kids and my leisure time Apple is after.

Apple has learned that an excellent user experience more often comes from tightly coupled software and hardware. It has also learned from iPhones and iPods that it can go a step further and subsume the computer beneath the consumer experience. Witness the amount of effort they have put into the UI. This tells me they aren't swinging for the fences on the first try, but attempting to build a platform they can splinter variants from in the future. The locked-down nature of many of Apple's products is disturbing, but new avenues will emerge if it is successful for a wider community of contributors to innovate. The App Store is controlled by Apple, but of course bounded problem spaces often yield better outcomes, and to date nothing else has come close. A true tablet computer may emerge over time (sooner, when its jail broken), but right now Apple is building on success, not giving the world another version of a failed experiment.

Smaller is Better

The Netherlands is releasing a fantastic new coin soon, continuing their tradition of stellar numismatic design. Not only is the execution brilliant, but the concepts supporting the design are deep, multifaceted, and engaging. Contrast this new coin with the EU-wide currency, and its clear that a smaller country (one with a strong design tradition) can produce much better results. The manner in which The Netherlands and the winner went about the design process are also worth review, as it was open and transparent as opposed to the committee-driven EU approach. US Treasury, please take note. Big companies don't do small things well, for a number of reasons. Institutional inertia is an obvious reason, but large companies also need to worry about their image and brand in the eyes and minds of consumers, investors, and competitors. Releasing product that is unfinished or has rough edges is something small organizations can do easily because they have little to lose. But small companies can also do things better - they can devote time, energy, and bring a sense of purpose and passion to a project because they have a direct connection to the outcome and the market.