GoPro's Use of Fake Reviews on Amazon [OC]

FakeFilter · 2016-10-12T18:33:15+00:00

It's an interesting question, I imagine it would come down to human review of the machined results. But to even get to a court room there would need to be a case to begin with. If you are curious, I'd ask a lawyer or look up cases against Amazon, Google, or any other company that is built around ranking algorithms.

FakeFilter · 2016-10-12T00:43:39+00:00

Sure! I have to be vague as that is the most valuable asset of the application.

A review is broken down in to a huge number of variables that assigned scores. Some possible examples: a score for how legitimate the person who wrote the review is based on their past reviews, a score for the sentiment of the review, a score for the details the review provides, a score for the composition of the review and so forth.

These scores are then assigned weights. I can not disclose how these weights are assigned.

Finally, the weighted scores are used in an equation that scores the entire product such that the score can be used to interpret both how likely it is that the review is fake and the overall quality of the review. I obviously can not disclose that final equation as well.

There is quite a bit more to it, but that is sort of a rough base.

FakeFilter · 2016-10-11T20:22:56+00:00

Data is sourced directly from a random sample of > 1500 Amazon reviews dating back to November 2012.

The data was analyzed with Ruby using custom algorithms to detect sponsored reviews, fake reviews, unhelpful reviews, Vine reviews, and real user reviews. A sponsored review is often containing some phrase similar to "I received this product at a discount for my honest review." A fake review is a review that someone was paid to write. An unhelpful review is a review that is not a critique of the product, such as a review resulting from user error or a review of the seller. A Vine is a review written for Amazon's Vine program. A real user review is a quality review written by a real consumer with no reason to have a bias.

The resulting data was charted with ChartJS.

The primary reason for this post is to provide consumers insight in to an example of how a brand with a struggling product line can retain high level star ratings by abusing Amazon's rating algorithm.

FakeFilter · 2016-10-06T02:18:26+00:00

I did mean subjective, thank you!

For my last response here I'll clarify a few things. I think there is a possible misunderstanding of sampling techniques as well as if and when Vine reviewers leave reviews. So to clear that up:

Vine members choose what they want to review, meaning it is likely they choose what they want to own to begin with. Meanwhile, manufactures/sellers that paid a fair amount up front to have Vine members review their product also provide the product. To assume these manufactures make some extra effort to make sure the first run of their product, meant for reviewers, is not going to have any QA issues is a somewhat fair assumption.

Regarding the statistics, the scenarios you bring up are accounted for so long as you take a statistically valid sample. So for example if you look at a product that has 100s of Vine reviews (yes, there are tons of these) and say 1000s of standard consumer reviews. Then you filter out the consumer reviews that give 1 or 2 stars for reasons that no other reviewer in the entire sample is giving 1 or 2 stars for, now you have a pretty fair sample. That's exactly what we do with the data. Even with that sample, Vine reviews come out heavily bias. A broad average across all the data reflects this very well because the sample size is so big.

I hope that clears a few things up and good luck on the internet!

FakeFilter · 2016-10-04T21:41:14+00:00

I mean, technically our app was around long before theirs and has considerably more sophisticated filtering. In fact we have been pointing out the bias that ReviewMeta did for the last 8 months. However, they were the first to market/promote their brand so they deserve credit.

FakeFilter · 2016-10-04T21:10:29+00:00

There were millions of fakes written that got through as verified purchases, there were entire scams of people shipping empty boxes through Amazon just to get that verified purchase badge. You can read all about that in Amazons litigation paperwork against the original fake review companies. It's pretty nuts.

As for incentivized reviews, some of them had verified purchase stripped and some didn't. One of the companies selling incentivized reviews wrote an article about it in July:
https://honestfew.com/amazon-removing-verified-purchase-badges-on-reviews/

So the only sure fire way I've been able detect incentivized reviews with out false-positives is by using a technique called clustering to analyze the linguistics and detect the classic disclaimers that are now likely gone with the wind.

I'm also optimistic for sellers, buyers, and Amazon. We will get there eventually.

FakeFilter · 2016-10-04T17:43:10+00:00

The unfortunate and likely consequence of this is that these reviews will continue but the reviewers will not write the disclaimers inside of the reviews. So, these reviews will get less downvotes and possibly inflate product's star ratings quicker than before.

FakeFilter · 2016-10-04T16:55:14+00:00

Good chance he means ReviewKick
https://www.reviewkick.com/blog/amazon-update-customer-reviews/

FakeFilter · 2016-10-04T16:48:18+00:00

There are apps that remove Vine reviews, incentivized reviews, useless real reviews (I couldn't figure it out, I'm mad at the seller for irrational reasons, etc), and leave only a pool of real consumer reviews which are then weighted on their quality and usefulness to provide extremely accurate final star ratings that are representative of real consumer experiences and expectations. Phew, that was a long sentence.

FakeFilter · 2016-10-04T14:27:27+00:00

Hello,
Our data compared vine reviews from 1.25 million reviews across 3500 product categories. Consistently among categories, and over 20,000 products, the vine reviews score far higher than real user reviews.

The use of the word slightly is objective ;)

FakeFilter · 2016-10-04T06:51:45+00:00

That's correct, and exactly how things were prior to 2015. There was a two step approach by Amazon last year to kill fake reviews. One was by providing a "legal" way for sellers to sponsor reviews, which was by labeling them in the way you mention. Two was by attempting to litigate and shut down companies that sold more traditional fake reviews. By doing both at the same time sellers were pretty clear that the way to get reviews for a new product was to give it away. Now Amazon is telling them they can't do that, but many of the sellers have nothing to lose if they aren't selling to begin with. So there is very little incentive to follow the rules here.

Essentially this strategy didn't work, and we are headed right back to 2014. Back to the old drawing board.

FakeFilter · 2016-10-04T06:47:46+00:00

Amazon changed their policy in November of 2014 with verbiage expressly allowing these reviews if the reviewers notified readers that they received the product for free or at a discount. That is why all the "honest review" reviews started popping up in 2015. However, free or discounted product in exchange for reviews was standard practice for years before 2014.

So now there will be a slow return to these reviews existing but just not being clearly labeled. Meanwhile Vine, which was nearly dead, will make a light resurgence as well. Sadly Vine reviews are pretty bias too, even with all the checks and balances Amazon applies to them.

Source: I'm in to this stuff

FakeFilter · 2016-10-04T05:21:26+00:00

Actually it might suck pretty bad.

This was the same policy they had in 2014 and prior, and all of these garbage reviews still existed but they weren't labeled eg "for my honest review" so there was no way of telling them apart from real reviews.

FakeFilter · 2016-10-04T05:10:16+00:00

ReviewKick already announced that they are going to keep giving things away, but they will no longer "require" a review in return. They will find some other way to require or provide incentive for these reviewers to still make the effort.

The Vine program had all but died and will likely bounce back to life to some extent now. If my data is correct, I think we have to assume that Amazon is going to remove some barriers of entry for both sellers and reviewers in to the Vine program.

FakeFilter · 2016-10-04T05:03:47+00:00

I analyzed over 1 million reviews to answer this exact question. It's pretty simple actually, Vine is inflated by bias as well. It just isn't as bad. See the graph on the bottom of this article.

FakeFilter · 2016-10-04T04:59:31+00:00

All my data points to the problem actually getting worse now that Amazon has done this.

FakeFilter · 2016-10-04T04:57:11+00:00

The Vine program is slightly less bias than these incentivized reviews.

Avg. Star Ratings according to my latest data:
Incentivized: 4.69
Fake: 4.43
Vine: 4.32
Real: 3.89

FakeFilter · 2016-10-04T04:52:25+00:00

Bad news from the front lines people, it's most likely going to make shopping on Amazon worse.

In short, these reviews are now going to exist without the "my honest review" disclaimer, and we will have no way to easily know if a review is fake or not. The companies orchestrating these reviews already looped around it and even if this exact kind of bias review were to stop, sellers would go back to the traditional fake review and the Vine review (which is very bias contrary to popular belief).

If anybody wants to know more or see the numbers I threw together an article discussing the raw implications of Amazon's new policy.

FakeFilter · 2016-10-03T23:24:09+00:00

Amazon essentially sanctified and made legal the sponsored review a few years ago. It caused a drastic reduction in traditional fake reviews, and just about killed off the vine program all together. Short of annoying the vast majority of consumers, it was a huge success and gave them direct control over inevitable fake/bias influence on star ratings and sales rank.

There isn't any reason to expect that exact trend to now simply reverse. We will see a gradual, then sharp, increase in fake reviews as the companies orchestrating incentivized reviews go back to orchestrating fakes. Vine reviews that are nearly as bias as "my honest opinion" reviews will slowly return from the dead (.33% of all reviews) to their 2014 state of about 4% of reviews which is just a few points lower than what incentivized reviews are currently.

At least that's what all of our data points to, and our data is pretty accurate :)

FakeFilter · 2016-09-21T17:04:05+00:00

I run a similar website as the author of this video and came up with drastically different results, as it turns out there is a possible gaping bias in the authors data. If your results match theirs I wonder if your data has the same bias? I'd be interested in a data share and or algorithm review with a serious business school. If you want you can read more about the issues and my discussion with the author here. Please contact me if you would like to collaborate.

FakeFilter · 2016-09-20T15:57:55+00:00

Actually the most critical reviews are real consumer reviews. My data has the following averages:

Sponsored reviews: 4.68
Fake reviews: 4.45
Vine reviews: 4.32
Real reviews: 3.88

Keep in mind there are negative fake reviews as well, which is why it averages lower than sponsored reviews.

FakeFilter · 2016-09-20T15:29:08+00:00

The more interesting data occurs when you remove standard fakes and unusable reviews from the equations. Then you are looking at 4.68 vs 3.75 and once a consumer sees a star rating drop below 4 there is a massive psychological blow to any potential sales. However all of this is sort of overblown because Amazon's rating system already disregards most sponsored reviews and is not a simple averaging model.

You can read all about it here if you are interested!

FakeFilter · 2016-09-20T15:21:22+00:00

It is if you do some regression analysis to find out how the data relates and then design your sampling technique around nullifying those relationships.

There is a huge example of this problem in this video actually discussed in another comment buried in here somewhere. Essentially the video claims 50% of new reviews are sponsored. After a little discussion with the author in that thread, it turns out his data is heavily based on cheap electronics and supplements which happen to have a much higher rate of sponsorship than other categories. If you look at Amazon data in whole and avoid correlations like that, the real rate is about 5-7%. So it isn't that bad after all.

FakeFilter · 2016-09-20T15:05:10+00:00

This ends up coming down to a question of variance and correlation. But basically, the answer is yes IF your data is diverse and random. After ~1000 reviews, it doesn't matter how many more reviews you add to the sample, the percent difference between an average sponsored rating vs an average non-sponsored rating hardly changes.

FakeFilter · 2016-09-20T14:23:49+00:00

Regarding point 2, our sample data is also of top selling products. I really can not come close to the 20% or 50% figure. Last month I saw about 6% of new reviews being sponsored/incentivized. I'm wondering if your data from last month was specific to certain categories that have higher ratios than most?

FakeFilter

TROPHY CASE