Companies have had varying degrees of success in implementing feedback systems that allow users to transact in a “trusted environment,” with eBay arguably leading the way. However, for a feedback system to be truly transportable and have any realistic kind of meaning, the basic rating mechanisms must be compatible… if not exactly the same.

This problem presented itself to me most notably while perusing the (now defunct) Yahoo! auctions site. As a embarrassing example of the expected behavior of their feedback system, my user rating is a “-1″. I guess in 1999 sometime (I honestly don’t remember doing this) I bought something on their auction site and never coughed up the cash… resulting in the seller rightfully leaving me negative feedback for the transaction. I had no other transactions through that site. So, the logic there is pretty clear: 0 positive feedback minus 1 negative feedback leaves me with a paltry -1 rating. Safe to say that, if I could and were to continue to transact the Yahoo! site, potential buyers and sellers should take notice of my rating and proceed with caution until I prove otherwise.

A clear example of how the Yahoo! system fails can be seen if you take a look at the following user rating here. In this example, while the user has exemplary feedback, the numbers simply make no sense. 38 positive feedback details from 27 unique users, no negative feedback, and an overall rating of 93209886?!? Now, anyone who has written low level code knows that this could be caused by a overrun of the integer containing the rating, but the math shouldn’t be that hard! 27 - 0 = 27!!! This is an extreme example, but I found these kind of inconsistencies all over the Yahoo! site. I found sellers with all negative feedback details and yet a miraculous positive overall rating. Something is certainly amiss and there is a proclivity for unscrupulous sellers to “game” the system.

The question becomes: how do we normalize these disparate rating systems so that the number makes sense both in and out of context? First, we need to have a fundamental understanding of what the underpinning algorithms for each site are, so obvious (eBay) others not so (Yahoo!). Second, there is a missing link in the chain that I think needs to be applied in order to “normalize” the reputation. A particular piece of feedback is only as valuable (information wise) as the person who left it is credible. If you have a stellar eBay rating minus one impossible to please buyer, then that one negative piece of feedback should have less of an impact on your reputation. Assuming there is truly a free market for reputation information, the impossible to please buyer should have a lower feedback rating based on their aggregate dealings with other seller. So, a true feedback score can only be generated in the context of who left each individual piece of feedback.

Over the top? Maybe. But it is your reputation at stake! Just as I breakdown and discount most of the information contained in the rantings of a loud and boisterous blogger such as Guy Kawasaki (no offense meant to Mr. Kawasaki), information of any kind can recognize its full value only in the face of complete transparency, which includes its source and their source(s), etc.. The 80-20 rule probably applies in this situation and helps us contain the work involved in generating the informational value of a piece of data, meaning that 80% of the time, one level of abstraction from the source should be enough to determine how credible the information is. It is the other 20% that present the real challenge.

Share This