The Fallacy of Reviews – Sample Size of One

Nathan S
by Nathan S

TFB’s reviews are typically the most popular articles and videos that TFB produces. I can say, definitively, that we take great care and pride in testing firearms for our readers. We always work to be as thorough as possible, covering the highs and lows of the latest firearms. We’ve done positive and negative reviews of product, always keeping the our duty to do right by readers and the manufacturers by telling it as it is.

Its important to note that a review of a gun is just that, a review of a gun. Unless otherwise specified of issues, returns, and fixes, we only go hands-on with a single firearm. Most times, the firearm works fine (a testament to the ingenuity and quality of most manufacturers), but occasionally we get a lemon.

Case in point, Military Arms Channel.

One of the first to complete a video on the new Remington RP9, the first video released about a week ago as a rather rough expose of a single unit having problems. Yes, Tim presented it largely objectively, but the negative exposure had a massive negative cascading effect on the handgun, especially with Tim’s previous (and very valid) coverage of the R51, problems which I duplicated personally.

Now why would so many other writers, YouTubers, and reviewers give generally good reviews if the gun was a piece of crud as Tim’s was? Simply put, because they had good guns. Yes, one may contend that Remington personally selected other media firearms, but many certainly were not from the Big Green marketing team.

Recognizing this and speaking to the fallacy of a sample size of one, Tim (to his credit) purchased and shot a second video on a second RP9. He found that the major issues originally shown were not present on the second handgun. Sure, he has seen some features or annoyances with the design that he goes into detail on, but find the handgun to be generally serviceable.

Moral of the story?

Always get additional opinions. Reviews of guns are only on a single gun and may not reflect the performance of the platform. A bad review does not mean the firearm is bad or that the article is truthful as it is a bad review. Don’t just latch onto the first or most boisterous opinion you find. Spend the time, do the research, and make informed choices.

Nathan S
Nathan S

One of TFB's resident Jarheads, Nathan now works within the firearms industry. A consecutive Marine rifle and pistol expert, he enjoys local 3-gun, NFA, gunsmithing, MSR's, & high-speed gear. Nathan has traveled to over 30 countries working with US DoD & foreign MoDs.The above post is my opinion and does not reflect the views of any company or organization.

More by Nathan S

Comments
Join the conversation
2 of 69 comments
  • CavScout CavScout on Feb 16, 2017

    On the new Rem pistol, I'm almost certain all the issues he had are MAGAZINE RELATED. They need to redesign the mag. This time, make it actually for 9mm and 9mm only.

  • Kalash of the Titans Kalash of the Titans on Feb 19, 2017

    I just think it's funny that Tim has done plenty of other "first shots" videos with quality pistols like CZ, Glock, S&W and has never needed a sample size of more than one. You know the old saying, if it looks like it and smells like it, well....

Next