Sorry for the long delay between posts. Fortunately the blogosphere has been providing ample reading material about the saga of the lost iPhone and the war of words between Apple and Adobe.
I’ve been doing some reading myself. Specifically, I just read F. Randall (“Randy”) Farmer and Bryce Glass’s recent book on Building Web Reputation Systems. Given that I’ve been thinking a lot about online reviews and reputation systems (e.g., this recent post), I wanted to hear what the experts had to say.
In the book, Farmer and Glass categorize the motivations for user participation as altruistic, commercial, and egocentric. Commercial motives are clearly the most problematic: a review site loses credibility if commercially motivated reviews are disguised to make their commercial motives. Most review site scandals arise from this kind of deception (e.g., this one, this one, and this one).
Sincerity is a necessary but insufficient condition for a review to be valuable to the person who reads it. There is still the “people like me” problem: sincere reviewers may still be uninformed, unreasonably biased, or may simply not share our tastes. User-generated content is an inherently subjective medium.
Given these challenges, it’s a wonder that online review sites work at all! And yet there are real success stories. My personal favorite is Amazon.com. While it has has its hiccups, Amazon nonetheless serves as a poster child for creating value by aggregating user opinions about products.
Amazon has a well-designed review policy that gets many key elements right:
- Reviewers have identities tied to purchasing history. That encourages disclosure (people use their real names) and discourages abuse.
- The reviews themselves–and even comments in discussion threads about individual reviews–are themselves reviewed as helpful or not. That may seem overly meta, but it does a lot to mitigate information overload.
- Grounding in objective information (product content, sales rank) reduces the ability to manipulate product perception through reviews.
The system isn’t perfect, but it’s good enough to be very useful.
But products aren’t the only reputable entities, to use Farmer and Glass’s term. What about service businesses, such as restaurants, gyms, etc. Or people?
If Amazon exemplifies online product reviews, then Yelp is the canonical example of a review site for service businesses. And, despite its own share of controversy, it is quite successful. But I dare say not quite as successful as Amazon. Part of the problem is that is demographics are less representative of the general online population (here’s what Quantcast says about Yelp and Amazon demographics for their US users). Also, there’s more variance in experiencing a service than in experiencing a product.
But Yelp has also has had a credibility problem regarding which reviews they allow to be published. Perhaps the root of this problem is that Yelp’s business model depends on paid advertising from the businesses reviewed on the site, while businesses would much rather have unpaid positive reviews. In contrast, Amazon makes its money buy selling products–which at least makes it perceived to be more evenhanded.
But neither Amazon and Yelp have touched the third rail of online reputation: people. LinkedIn dabbles in this space by allowing its members to review one another, but reviewees have veto power over reviews–making the review graph more of a mutual admiration society.
A recent startup, Unvarnished, is trying to create a review site with teeth. Farmer argues on his blog that Unvarnished is breaking some major rules:
- It displays negative karma–that is, it allows people to write negative reviews of one another and displays those reviews.
- The reviews are not clearly tied to context (e.g., were the reviewer and reviewee co-workers?).
- The anonymity of reviewers does not incent altruistic or even egocentric behavior, and is thus a recipe for abuse.
I’m not as down on Unvarnished as Farmer, but I agree it will have an uphill battle to succeed. Ironically, for all of the public concern about Unvarnished becoming a trollfest, the reviews skew strongly positive. This is probably an artifact of how Unvarnished is growing its membership: current users ask friends to review them.
I agree most with Farmer that Unvarnished’s incentive structure seems problematic. A person’s friends will probably be inclined to write positive reviews, and may even be annoyed at having to write them anonymously. A person’s enemies may be inclined to write negative reviews as a form of attack or revenge. But it’s less clear what will incent people to write accurate reviews–or what will signal to readers that a review is trustworthy.
All in all, I think that these are early days in the online reputation space, and that there is ample room for innovation. Facebook’s recent release of “like buttons” is an ambitious attempt to boil the ocean of “social objects”. A best poster award at the recent WWW 2010 conference went to Paul Dütting, Monika Henzinger, and Ingmar Weber’s “How much is your Personal Recommendation Worth”. Hopefully all of these attempts to research and innovate will lead to a world where we can derive real value from others’ opinions and feel incentivized to contribute our own.
