A "minor" college class project intended to demonstrate how first-time users fared in doing basic tasks on different smartphones has triggered an Internet wave of mockery, condemnation and invective. In a 10-minute video, the iPhone 4 and the Samsung Focus running Windows Phone 7 are rated superior to the Android-based HTC Thunderbolt and a RIM BlackBerry Storm.
The video was originally posted on YouTube this week by the students who made it, as part of a project for a Harvard University summer course, "Human Factors in Information Systems Design." It was later made private, but then reposted on YouTube from cache by Surur at WMPoweruser.com. His accompanying post had almost no details on the video's background.
SMARTPHONE COMPARISON: 4 reasons Windows Phone 7 will beat iPhone and Android
The class is taught by Dennis Galletta, who is a professor of business administration at University of Pittsburgh. His main research areas are, according to his biography page, "human-computer interaction, with specific attention to user attitudes, behavior, and performance," and his teaching interests include human-computer interaction and electronic commerce. He's been an adjunct professor for the Harvard summer program for several years.
The six-week summer course focused on human factors and corporate website design. Galletta gave the students a menu of related class projects, including a usability comparison of the four smartphone platforms. That's the one they chose.
The result, and intent, of the project was a "fun initiation into usability testing but NOT a definitive or reliable test of the platforms," Galletta says. His project requirements were to "compare the usability of the four platforms in an objective way, with points taken off for being subjective" and "squish it into 10 minutes or less [on video]."
The project was simplicity itself: All the students owned smartphones and they lent them to each other to test the three tasks. Each tester had no previous experience with that phone or its UI. The intent was to see whether and how users could figure out each task, without any familiarity with the phone or operating system and without any instruction or guidance.
Each user was videotaped performing each task in sequence and the resulting video was edited to keep it the 10-minute limit imposed by Galletta. The on-screen narrator is one of the students involved in the project.
In the video, the ratings for each task on each phone seem loosely correlated with the number of touches and taps and the length of time each task takes. But the actual criteria are not spelled out. The demonstration is organized by task, so we first see each user trying to make a phone call, then in the same order trying to add a contact, and finally sending a SMS message. Each separate test by the different users shows a close-up of the phone on a flat surface, and the hand or hands of the user.
Taken at face value, the video is most intriguing when one tries to follow the thought process of the novice user, as revealed by his or her touches and taps to the phone's screen. For example, the first test shown is making a phone call with a Samsung Focus, running Windows Phone 7. The user taps correctly on an icon, or "tile," with the image of a phone's handset. But what next appears is the "History" page of recent calls. The user apparently doesn't readily see how to make call from that page, and presses the home button to go back to the homescreen. After trying out a number of other actions, the user taps the phone tile again, finds another soft button at the bottom of the history page and is finally able to type in a phone number and make the call.
That experience won the Samsung device 3 out of 5 possible stars. The iPhone 4, the BlackBerry Storm and the HTC Thunderbolt were all much faster, with just one or two touches needed to bring up the dial pad.
Adding a contact was better for the Samsung device, which earned 4 stars; the iPhone earned 3. The users of the Storm and Thunderbolt struggled for much longer to decipher the interface and the BlackBerry user actually failed to add a contact.
The iPhone and Windows Phone Samsung handset tied with 4 stars in sending an SMS message, the Storm got a 3-star rating, and the Thunderbolt came in last.
What's striking about the video is the contrast in user actions: Often they're actively scrolling, apparently searching for something that signals clearly to them what the next step in each process should be; at other times, they're often almost completely inactive with their hands, apparently studying the screen to decipher or interpret, and sometimes trying out the various icons or soft buttons presented to them.
The unscientific conclusion for the project: Novice users found iPhone 4 and Windows Phone 7 on the Samsung Focus somewhat more intuitive to use in the three tasks tested.
But for a legion of bloggers and commenters the project is an example of what's wrong with higher education in America. The denunciations fell on both the video comparison -- lame, stupid, fake, horrible, poorly executed, a conspiracy -- and the phone testers -- moronic and stupid.
"I find it quite concerning that such a supposed high brow university would throw together such a poor and unbalanced experiment ..."wrote Jay Oakesey, at Time's Techland blog, which offered Harry McCracken's Technologizer.com account of the video.
At WMPoweruser, one commenter, Tablewriter, wrote, "the research is so flawed that the guy deserves to lose his job over it. It's what I'd expect from a fourth grader. There is NOTHING of value here."
"Clearly this was RIGGED," wrote jwd0808 at YouTube. "The HTC Thunderbolt's home screen was not the default -- they removed both the people and the text messaging icons. They guy using the Windows phone was just an idiot ..."
Plenty of others joined in attacking the testers. "I think some of those testers are freaking stupid :|," posted cPTcAPSLOCK, at WMPoweruser.com. "I never knew people could be so dumb," posted Bloodoathjg on YouTube. "[A]nyone with half a f***ing brain can navigate these phones," wrote a user with the handle dicksoper.
More than few assumed Galletta was the narrator: He was excoriated for calling the iPhone the "iPhone 4G," though some posters said he probably meant "fourth generation," and the Samsung device a "Windows 7 phone" instead of "Windows Phone 7."
There were more considered responses. "Of course, the experiences of a handful of clueless newbies aren't a definitive verdict on these operating systems," wrote Harry McCracken, in his Technologizer blog.
(McCracken also updated his original post to note that this was a class project by students, not a research initiative by Galletta. "Personally, I'm less interested in how well neophytes fare with a phone than how efficient and intuitive it is for a more experienced user over the long haul.")
Galletta told his students the video could be made more useful by using a larger number of tasks and users, and a more scientific measurement process, or at least a documented one.
John Cox covers wireless networking and mobile computing for Network World.
Twitter: http://twitter.com/johnwcoxnww
Email: john_cox@nww.com
Blog RSS feed: http://www.networkworld.com/community/blog/2989/feed
Read more about anti-malware in Network World's Anti-malware section.