Interview with Ben Shneiderman, by Amal Dar Aziz, May 2008

Ben Shneiderman’s natural inclination towards visual thinking ignited an obsession with information visualization. Father of many visualizations, including the Treemap, Shneiderman has a long history as a researcher and a professor. He continues to inspire designers to create tools that aid creative expression and that help people become better visual thinkers.

AZIZ: What first brought you to computer science, since I know you did your undergraduate degrees in mathematics and physics?

SHNIEDERMAN: Right, well when I started, there was no computer science, but I was fortunate to work for a group of physicists when I was an undergraduate at the City College in New York, and I began doing bubble-chamber 3D track reconstruction programming for them. And that gave me a terrific experience, and it was fascinating to me so that going to classes got in the way of my education. I was learning a lot of things that were just terrific, so I was really engaged with it. I was fortunate also to have Richard Hamming, a famous computer scientist, as my professor for two semesters, in which he taught only the things he had invented. And he was an impressive character who got me moved along from the low-level notions of programming, to the higher-level issues of design of algorithms. His famous saying that "the purpose of computing is insight, not numbers" has stayed with me very strongly, and I now say that the purpose of computing is insight, not pictures. And that relates to information visualization, which has become my obsession over these last twenty years.

AZIZ: Tell me bit more about your "obsession" with information technology. Did earlier work studying programmers influence the direction of your work in this area?

SHNIEDERMAN: Well, my background was always very visual, starting with photography. My uncle, who was a famous photographer named David Seymour, and I was very visually-oriented, considering to be a photographer myself. I chose to keep photography as my hobby, and made computing my profession, which has worked out just fine. I still enjoy working on photography applications in computing, and building websites on my uncle's work, so that has kept me in a happy blend of disciplines, which I've always liked, rather than the narrow specialist view. So with an orientation towards visual thinking, it was quite natural that I would begin to think of programming as a visual task.

One of my early contributions was known as structured flow charts. They're commonly known as Nassi-Shneiderman charts, after me and my colleague Ike Nassi, but they were widely used during the 70s, 80s, and 90s, and it's a great story itself. So I was already thinking visually, and the innovations I began to develop in 1990s, like the Treemap, and what we call the Startfield display, were a natural extension of my way of working.

AZIZ: Tell me more about the story behind the Treemap.

SHNIEDERMAN: The Treemap story is a wonderful story, which continues to this day, after almost twenty years. It started from my frustration with managing the hard disk in our laboratory, where fourteen of the students and staff were sharing an 80 megabyte hard disk on a Macintosh. Seems small today, but at the time that was generous! So fourteen shared users, and when it filled up, I had to go and find the person to bother to remove their files. So, finding out who was the major consumer of the filespace was a tedious job — I had to look in each one of the folders. And I was looking for an overview that would show me in a visual picture what was on the hard drive. Not by tracks or sectors, but by directory structure. And I wanted more than just the number of bytes in each of these major fourteen folders — I wanted to see the substructure. I wanted to see who had programs there, who had large data sets, or large images. And where were the opportunities to reduce the space utilization.

I struggled with that for many weeks and months, drawing all kinds of diagrams, with rectangles and circles, and triangles, and other shapes. I was trying to make a recursive structure that would work for arbitrary depth. I had this "ah ha!" moment one day in the faculty lab. I was working on something else and it just struck me that there was a way to do this recursively, in a way that became known as "slice and dice." And so that was the first algorithm for doing treemaps. It took me three days to get the six lines of recursive code just right, but it really seemed to be the right idea. And that was implemented soon by Brian Johnson, a PhD student who made his PhD on that project.

We had many other implementations, and other the years I was just wonderfully surprised when other people came along and found different ways of doing a Treemap layout. Martin Wattenberg's Clustered Treemaps were used for SmartMoney.com's Map of the Market. Jarke van Wijk's Squarified Treemaps became the more popular approach because they used more square-like boxes that have a cleaner aspect-ratio and easier to point at, and aesthetically-attractive. They gave up the sequential ordering of slice and dice, which was something I hadn't even thought about doing. My mind was open to these other possibilities, and then we added two more designs. So we have at least five major algorithms, and then many variations. The great fun when you have a new idea, a good idea, is how other people take off on it, and build extensions that you just never thought of, that aren't so clever. That's been the continuing history of this topic. Every week or so I get an email from somebody who says: "what do you think of this?" They're often running with another great idea, so the widespread acceptance is one of the things you're looking for when you have a research innovation, and that has been very satisfying.

AZIZ: What's the new direction visualization technology is headed? What are the challenges, and where do you see information technology fitting into peoples' everyday lives?

SHNIEDERMAN: In the short term, the problem of visualizing networks has been an important driver for our current work. We have two major projects: one of them you mentioned is SocialAction, the PhD work of Adam Perer, that has a wonderful approach to integrating statistics with visualization. There are many tools that do statistics on networks, and there are many tools that do visualization, but the integration has been poorly handled, making it clumsy, I would say. It takes a long time if you want to make certain adjustments: to the display, re-calculating the statistics, redisplay, re-calculate, and it takes minutes rather than seconds. So SocialAction provides powerful statistical algorithms, integrated smoothly with the visualization. Giving strong user control to select portions of the network, focus on those, identify important nodes or links and understand the dynamics of these networks. So that been a good success story.

The second project is the PhD work of Aleks Aris, and it's called NVSS — not a charming name, but it's Network Visualization by Semantic Substrates. And this depends on the rather contrarian view, which is that you shouldn't lay the nodes out. First, put the nodes where they belong, don't let them float around, but put them where they belong in a meaningful display that might have multiple regions. So if we're looking at Senate voting patterns, we might have the Democrats in one region, and Republicans in the other regions, and that is an understandable pattern. We might lay them out by, for example, their seniority, or we might put them on a map based on the state they're from. And so those meaningful layouts are a good start, and then we give users control over the link visibility. So instead of cluttering the display with thousands of links that it makes hard to make sense of, we give the users control so they can just display the dozens or fifty links that might be relevant to their task at any moment, and be able to follow every link from it's source to it's destination. So, in the short term, we're polishing these and applying them, and learning to make them effective for a variety of applications. For example, now we're working for the National Cancer Institute, to help them understand the partnership relationships among the 900 organizations that they work with across the country, and how these groups collaborate with one another. SocialAction will play a major role in that analysis, and it's the first time they've been able to do that kind of visual analysis.

AZIZ: And where do you think the common everyday person fits into your ideas of visualization tools?

SHNIEDERMAN: There is a growing utilization of information visualization by broader audiences. I won't say common because nobody is common, everyone is special, but but I think there's a growing number of audiences which are finding these tools useful. Some of them are more technically-oriented, like economists, scientists and analysts. So they will take the time to learn the new tool because it gives them a new power. In the same way that learning how to operate a telescope or a microscope is a new power, learning how to operate the visual tool gives you new powers. We see the progression in different professional fields, and it's a great satisfaction. Some of the people have made a leap towards the wider public. I'm especially impressed by the New York Times, and the quality of their illustrations on a daily basis. And increasingly their interactive representations of these ideas. They can research ideas — our work and others — and put it in the mass media in a wonderful way. And in their own skills and capabilities, and knowledge of how to present information, to make these even more effective than we might have dreamed of. So I do see that happening. I must say that I thought that the Treemaps were visualizing the content of the hard drives that I started with would be on everybody's hard disk in two years. But, it takes a while. New ideas, and especially visual ideas, don't always spread rapidly. It may take a while.

AZIZ: Why do you think that's the case?

SHNIEDERMAN: It's something new, it's something different. Not everyone is visually-oriented. I might ask what percentage of the populations owns binoculars or a telescope? Or microscopes? It's a small fraction of the population, but for them it provides a powerful way to see the world and make discoveries. So that, I think, is the realistic assumption, is that i hope it'll broaden further, but it will take it's time to grow out. I think we have been raised in a world for the past century that has had more of an orientation towards logical, numerical, tabular, spreadsheet kind of thinking. And the shift towards visual and intuitive, and pattern-recognition is a historic change. And it will take a century for that pendulum to move back to the way it was. In Leonardo's day, people understood the visual cues and literacy. Modern students or adults at Leonardo's paintings, like say the Last Supper, they see a group of people at dinner. But the people at Leonardo's time understood that a hand raised, lowered, pointed upward, or downwards, hand opened or closed, all meant something very different. And so they were attuned to those visual cues, and I think we will go through a new process of education, so that the next generation of students and adults will be more attuned to these visual opportunities.

AZIZ: You mention Leonardo, and I know Leonardo is one of your muses. Why Leonardo in particular?

SHNIEDERMAN: Many people that read about Leonardo knew about his famous paintings. As I began to read more and more about him, I became more charmed and impressed by what he had done. We was wonderfully the Renaissance man we all dreamed about. But he understood these different disciplies. He made contributions to them. He was a colorful figure. He was famous for drama, and music, and performances, theater, and the quarters of Lorenzo de' Medici. And then there are the stories of him going to the markets in Florence, and buying birds meant not to be eaten but instead setting them free. He was a vegetarian. He had a very drawn set of principles about his life. He was courageous in that he participated in some thirty dissections of cadevars, a very risky thing to do in his day, so that he could learn about the anatomy of the human body, and drew them in a way that was never seen before. His drawings are so detailed that you can see the flow of circulation from the heart, lungs, and through the veins and arteries. But he didn't quite get the idea that blood circulated back and forth. The belief at the time was that blood went from the heart, out through the arteries, to the tips of your fingers, and then vanished. It took another 150 years for William Harvey to make that interpretation about circulatory flow. But Leonardo's drawings are just at the edge of making that discovery. Now many people believe that if he had made those drawings public, which was not possible during his day, that he might have accelerated the process of medical discovery. So again and again I came to find the stories of Leonardo impressive, amazing, and satisfying.

On the other hand, his was a difficult life, and I wouldn't have traded. He had a lonely life, never married, never had children, and had a limited life in the way we might think of human relationships. He had a troubled childhood — so those are the tradeoffs that we make in life.

AZIZ: So how to we add creativity when interpreting the world around us?

SHNIEDERMAN: The tools that students have today are rather remarkable, and they offer new opportunities. I think the best example is the success of YouTube, and the fact that every, or at least most, high school kids can edit their own movies and create animated films, and put them on YouTube. That's a remarkable transformation. Thirty years ago, only the Hollywood studios could make that kind of film. And now we see it's broad participation in a wonderful way. So that's an example, of what I see of technology — it was designed to enable more people to be more creative, and more often, which today, seems like the next gold. We've spent 40 to 50 years in the computing field working on productivity support tools, where the measures of success are easier to see and the financial payoffs are more direct. Now, more people are devoting themselves to creativity support tools, where it's much harder to measure the outcomes and demonstrate the value. We're seeing the creativity that comes from powerful tools like Photoshop, and Dramatica Pro for screen writers, and one after another. And of course, in the sciences as well, the visualization tools that we've worked on have led to discoveries. It's very satisfying to see the feedback, read the feedback, of people who have used SpotFire and other tools, and report the discoveries they've made by having such powerful tools. These happen as a result of some good design, but we didn't have a systematic and thoughtful plan for taking this on. And now I think we're getting closer, to understand how to broaden that success story from one tool or one domain to a wider variety of disciplines and tools. I see great opportunity for making more people more creative, more often, which I think is an important goal.

AZIZ: Tell me a little bit more about what you're doing these days.

SHNIEDERMAN: My provost tells me I have the best job in the world, which means as a professor, with moderate administrative responsibilities, I have the chance and the pleasure to work with students. The grad students I have are wonderful, and I enjoy working with them, I'm paid for the opportunity.

I mentioned two projects, which are network visualizations, but the next major project has to do with electronic patient records research. We had years ago devoted tools to visualize the patterns of one patient's history, but now the challenges are if we have 100 million patient records histories, how do we search them to find patterns of disease, or treatment, or successive outcomes? If I want to track patients who have a certain type of tumor, they're receiving tumor therapy, or surgery, or radiation treatment — and what outcomes can I choose? Can I find those patterns for a rapid recovery, or can I find patterns for diabetic patients receiving medication X, having side effects Y, provided within a two-month period. Many physicians and clinicians and researchers are looking into these data sets to be able to understand patterns of disease and successful treatment, but the tools, such as SQL query fomulae are just too difficult to use, and so we developed a variety of tools, both with the Washington center and the Harvard i2b2 project, that enable our collaborators to find patterns of patient outcomes within these large datasets. It's a real challenge because the data is complicated, the data is messy, it's missing data and has redundant data, and it really puts us to a strong test. I'm really satisfied with the progress, which has taken us many years to get to this point. When the tools are good enough to give to our collaborators — I wouldn't say to broadly disseminate, but working with them we understand but works for them, and what doesn't work. And I'll go back and fix it. We respond to the requests. We've always been successful by working closely with real users with real problems. Not our fantasies of problems.

AZIZ: What was it like starting the HCI lab at the University of Maryland?

SHNIEDERMAN: As you can tell, I always had a somewhat rebellious streak, and was doing different things, diverse things. I was very much influenced by Marshall McLewitt, who made clear that specialization was the old way, that the global village and the electric world broke down these barriers and allowed interdisciplinary collaborations, and even required them. Well, it's not quite as good as all that, but I always was attracted, because of my diverse background, to spanning these disciplines. Physics, poetry and psychology all seems to be quite close to one another. I was naturally attuned to this notion of the psychological aspects of programmers initially, and I began to study how programmers worked, and what enabled them to be more successful. That lead to a book in 1980 called Software Psychology, that talked about the marriage of these two disciplines.

By 1982, I was getting more interested in trying to take this idea and establish it beyond my own work, and I was very pleased that one of my senior colleagues and very good friend, fames and respected professor Azriel Rosenfeld, one of the leaders in computer vision, was starting a laboratory that was called the Center for Automation Research. And we wanted interdisciplinary groups on his computer vision group. He recruited a colleague to start a computer robotics lab, and he asked me to start a human-computer interaction lab. So this was great, because this accelerated my agenda by at least 5 years, to get the participation in a larger organization, and the respect of Azriel Rosenfeld gave me the opportunity to reach out across campus to other departments and to collect a group of people — like-minded people — who shared that vision.

Our early work was on hypertext systems. We were building systems for the Holocaust Museum in Washington DC, and it's working for them that we developed the idea of the interface with a link. And that certain words in a paragraph are highlighted, you click on them, and you go somewhere else. It was very satisfying to develop that idea. We did about 15 experimental studies for different ways of arranging that, and demonstrating the value as opposed to the typical way at the time, which would be a paragraph of text, and then a numbered set of choices, where you would then press a number. And we did show that if you wrote your text with those opportunities embedded in it, and highlighted in a light way — we tried bright red, which caught people's attention but interfered with their reading. And so the light blue which we chose worked very nicely, and that's how light blue came onto the red. So we produced the world's first electronic scientific journal, which was the July 1988 issue of Communications of the ACM, which was the proceedings of the Hypertext 1987 conference, which we did as an electronic publication. And then in 1988, did the book Hypertext Hands-On!, which was the first electronic book in the Library of Congress. And then by Spring 1989, Tim Berners-Lee cited that work in his manifesto for the web. He was clever enough to change the lingo — we had used embedded menus, and he called them "hot links." And that certainly stuck.

AZIZ: Tell me more about your interest in photography.

SHNIEDERMAN: I was interested in photography, and knew about about my uncle's work with Magnum Photos and had access to that as a high school student. I entered photo contests. As a college student I was the photo editor for the yearbook for three years. I really worked hard on that. But my choice was to go to computing, which was a great attraction. And I chose that. But I've over the years taken on many projects, such as our work called PhotoFinder, which was a tool for managing your photos, to allow drag and drop annotation, which became a patented idea. And that's led to Bederson's PhotoMesa, a better version of a photo management tool. So those were satisfactions, and resulted in published papers, and student projects along the way. We worked on websites to exhibit my uncle's work, for the International Center of Photography. The 1996 retrospective on his work was matched by a web exhibit that we prepared — it's still on the web. It's one of the early and still most visited photo exhibit website. More people visit that photo web site each month than saw it during the entire time the exhibit was open. The physical exhibit was open in 1996, though it has a remarkable power, and lives on, and enables people to tell stories. So when I've arranged exhibits of my uncle's work, at the Corcoran Museum in Washington DC, George Eastman House in Rochester, New York, or the DeYoung Museum, most recently, in San Francisco. I've worked with the curators to make a very interesting web site in each case. It's been a lot of fun for me to help guide them to thinking how to present photography on the web.