4th Grade Spelling Bee
One comment that caught my eye was that of David C. Mason in response to my post “Launch Update + Comments on Avvo Rating.”
David wrote:
“Despite your front page claim, your site doesn’t appear to analyze the attorney supplied data in any objective manner. As long as these folks place as many lines in as many of the fields as possible, their score gets adjusted upward. I can put “4th grade spelling bee” in the award section or Island of the Blue Dolphins Book Report in the speaking engagement and your program will accept it and adjust the score.”
Thank you for your post, David. I understand how you could reach this conclusion at this early stage of our product, but it is incorrect. As such, I want to make sure that everyone understands how our profile scoring works for this kind of data.
When lawyers claim their profiles, we ask them to supply entries that demonstrate their recognition from the legal industry. In requesting this, we are expecting that people will not put in random items just to inflate their Avvo Rating. Based on this expectation, the model’s scoring system temporarily credits certain “unknown” information with points pending review of the submission.
Allow me to explain: When someone uploads an award, for example, our system tries to find that award in our database. If the system finds the award, the system immediately gives the award the number of points assigned to that award under our scoring guidelines. If the system does not find the award, it is marked as “unknown” and the system immediately gives the award the minimum number of points that any award can receive. This “unknown” award is then forwarded to a team that reviews the award and assigns a value to it according to our scoring guidelines. If the “unknown” award in our example is worth more points than the minimum, additional points are assigned to the lawyer’s Avvo Rating. If the “unknown” award is not worth any points like David’s spelling bee or some other award that is not relevant to the lawyer’s body of work - then the assigned points are subtracted from the lawyer’s Avvo Rating.
So, theoretically, a lawyer could come into the site, claim his or her profile and fill it with a bunch of silly entries in order to inflate the Avvo Rating. But, this inflation is short-lived because it will go away when we map this new and unknown data.
I don’t expect this to be a big issue, because attorneys better than anyone understand the issues here. Still, I wanted to make sure that I responded to David’s post to ensure that everyone understands our scoring procedures.
Mark
June 14th, 2007 at 3:32 pm
“So, theoretically, a lawyer could come into the site, claim his or her profile and fill it with a bunch of silly entries in order to inflate the Avvo Rating. But, this inflation is short-lived because it will go away when we map this new and unknown data.”
This is hardly an incidental post - this type of issue was highlighted in John Cook’s PI blog just a little earlier with regard to the class action filed against Avvo today.
June 14th, 2007 at 8:07 pm
Mark, I completely agree with you.
All rating systems at the end of the day are fallible. Would you expect Zagat to completely predict whether or not a given restaurant is right for you? No. Just because a wine scores high in Wine Spectator doesn’t mean you’re guaranteed to enjoy it. Just because Google shows a few URL’s first doesn’t mean they are the best ones (or even contain accurate information). But are they better than no information at all? You bet.
Anyone who thinks they can do a better job creating a rating system in an opaque industry is welcome to start a competing effort. And people are welcome to critique various rankings. But enjoin such a service? This smacks of someone being very, very afraid that some new light is being shown into a dark space where none existed before.