The Fallacy of Reviews – Sample Size of One

TFB’s reviews are typically the most popular articles and videos that TFB produces. I can say, definitively, that we take great care and pride in testing firearms for our readers. We always work to be as thorough as possible, covering the highs and lows of the latest firearms. We’ve done positive and negative reviews of product, always keeping the our duty to do right by readers and the manufacturers by telling it as it is.

Its important to note that a review of a gun is just that, a review of a gun. Unless otherwise specified of issues, returns, and fixes, we only go hands-on with a single firearm. Most times, the firearm works fine (a testament to the ingenuity and quality of most manufacturers), but occasionally we get a lemon.

Case in point, Military Arms Channel.

One of the first to complete a video on the new Remington RP9, the first video released about a week ago as a rather rough expose of a single unit having problems. Yes, Tim presented it largely objectively, but the negative exposure had a massive negative cascading effect on the handgun, especially with Tim’s previous (and very valid) coverage of the R51, problems which I duplicated personally.

Now why would so many other writers, YouTubers, and reviewers give generally good reviews if the gun was a piece of crud as Tim’s was? Simply put, because they had good guns. Yes, one may contend that Remington personally selected other media firearms, but many certainly were not from the Big Green marketing team.

Recognizing this and speaking to the fallacy of a sample size of one, Tim (to his credit) purchased and shot a second video on a second RP9. He found that the major issues originally shown were not present on the second handgun. Sure, he has seen some features or annoyances with the design that he goes into detail on, but find the handgun to be generally serviceable.

 

Moral of the story?

Always get additional opinions. Reviews of guns are only on a single gun and may not reflect the performance of the platform.  A bad review does not mean the firearm is bad or that the article is truthful as it is a bad review. Don’t just latch onto the first or most boisterous opinion you find. Spend the time, do the research, and make informed choices.



Nathan S

One of TFB’s resident Jarheads, Nathan now works within the firearms industry. A consecutive Marine rifle and pistol expert, he enjoys local 3-gun, NFA, gunsmithing, MSR’s, & high-speed gear. Nathan has traveled to over 30 countries working with US DoD & foreign MoDs.

The above post is my opinion and does not reflect the views of any company or organization.


Advertisement

  • Chuck E

    So where’s your disclaimer that this post was sponsored by Remington? 😉

  • junyo

    The sample size argument is a valid one, however the problem that the variability of samples presents is one of quality control. Any random factory produces outstanding examples and crud by sheer random chance. The question for the consumer is a) what proportion of their output falls into the Outstanding-Acceptable range, and b) is their QC process good enough to insure that when I pick up an example from my LGS it’s likely to be one of those guns and not the crud. The range of experiences/reviews gives you at least some data points in that regard.

    For me, hearing about how ‘great’ Gun Company X’s service is functions the same way; in a perfect world, I never want to deal with return/service on a gun*. If they’re really good at getting a gun running, you have to ask why their processes aren’t insuring that the gun is coming that way in the first place.

    *Exception for standing behind a product that needs help after severe/extreme use; they fix a gun that’s fallen out of a helicopter that’s a net positive.

    • Paul White

      I don’t know, I do care about customer service *because* all factories produce lemons.

      My brother and I have both sent a gun back to Ruger; his was a 10/22 that wouldn’t go into battery, mine was a 9E that wouldn’t get through a magazine without multiple failures.

      I’m not knocking Ruger; 1/2 my handguns came from them, and my brother’s got 3-4 of their riles as well as revolver from them. But it does make me more likely to buy from them knowing that ont he off chance I did get a lemon they’ll fix it.

      • junyo

        I hear you, and I’m not saying great service is a disqualifier, I’m saying volume of service is. If everyone who loves the gun/brand tells you how awesome their service is, that tells you a lot of guns are leaving the factory not working. Example, Keltec. I own Keltecs, and I’m not going to crap on them for what they do well. But at the end of the day, if you hang out on Keltec forums, a decent percentage of the people there have had to send their guns back. Heck, the first gen RFB was basically more useful as a club than as a gun out of the box (I kid, I kid.). Did Keltec make it right? Yup. Do I need that kind of aggravation in my life? Nope. Service is a consolation prize when QC has failed.

      • koolhed

        The last NINE Ruger firearms that I bought had issues that should have kept them from going out the door.

        • Hoth

          Details?

        • Bimberle

          Dude, you’re stubborn. 1 lemon, ok. 2nd, maybe. But 3-9? Wow!

        • junkman

          20 Rugers, ZERO issues.

          • gordon

            For me it was four Rugers, three lemons. The 4th is a 4.2″ SP101 that is quite satisfactory even if a little sloppily fitted. I kept that one. We may have different standards. All four of the Rugers I bought were quite functional they just had things like front sights at obvious cants and seams that didn’t match well. And all needed action work to make smooth.

    • nova3930

      All factories do produce lemons. The ones with better processes in place will generally produce far fewer though. Striving for six sigma and all that…

      • Bjørn Vermo

        If they have good QC, you will never see the lemon. If they have good QA, they will have good control of the factors governing their quality. Both of these cost “overhead” and are difficult to implement properly.

        • Vhyrus

          Quality is free. Shoddy work is what costs companies money in the long run.

          • Gregory Markle

            Not really disagreeing with you as shoddy work is a huge issue but in a manufacturing setting quality is NOT free in any way, there are costs associated with it. It’s always been my opinion when working in quality circles that if it’s determined that the product will not bear the cost of ensuring quality it’s probably not worth the cost to tool up for production in the first place. In the long run you just end up spending the money in customer service and support while losing customer confidence (which is a priceless commodity you can’t buy back.)

        • Zachary marrs

          Good is not the same thing as perfect.

        • gordon

          This is why I just spent the money and got a Korth this time. I got sick of QC problems from Ruger & S&W. I am quite nit-picky, and I suspect that lemons from both a rare, but they seemed to find me. I have not found anything that bothers me on the Korth so far. Korths are probably not worth the price for most people, but for me they are. Even small quality issues on a gun bother and bother me more the longer I own it. This one seems like it will be stress free.

          • junkman

            Well, if want a Korth, all well & good. I have, from personal experience, lost all confidence in German products.

          • gordon

            You have had bad luck even with German firearms? That sucks. What I don’t like about Germany is that is and pretty much always was largely socialist.

  • Edeco

    Small sample sizes happen. In this situation a reviewer getting a lemon is rightly interpreted as a fairly strong bad sign. I forget the proper terms, but if 5/5 performed well it would be a weak indication that the gun is good, if only 4/5 perform it’s a stronger indication that it’s bad. Might not seem fair but pfft, this is big-boy time.

  • Joseph Goins

    Except when the one that doesn’t work is yours. If I bought a dud (especially on a new platform), I wouldn’t go out and waste my money on another.

  • Stephen Paraski

    No. The “Moral” of story is Remington does not care. Case in point, Trigger System. Google how many hundreds of millions Remington has spent in Lawyer and Court fees defending a known, faulty design. And you thought new owners would turn it around?

    • Sunshine_Shooter

      That’s a different story.

    • Wow!

      Like sunshine shooter said, that was actually MSM propaganda to open up a legal route to ban all firearms with “faulty triggers”. Their idea was once the hole was opened, anyone can just discharge their firearm, call it faulty, and then have the gun banned from the market since it is basically impossible to prove if every trigger that is produced from a factory will not produce the malfunction or if the malfunction won’t happen on the trigger in question even if it isn’t malfunctioning now. It is what got CA’s as well as other state’s “drop tests”. Remington did have a trigger recall later, but that was a separate issue and trigger recalls are very common in the firearm industry.

  • Fuddly

    Quality control and quality assurance are both aspects of a production process and every failure that is not identified is a testament to the lack of attention paid at this stage, regardless of whatever failure rate the process that produces the good yields.

    While I question MAC’s testing a bit, namely considering his two brands of ammunition “fine” simply because it worked in other firearms (I can find a dozen guns that shoot Tula if I tried), the failure on the first pistol still reflects poorly on Remington’s QA/QC and his review(s) are still an entirely valid representation of a scenario a consumer could expect to face.

    Unfortunately for the consumer, I doubt they have the clout of an gun e-journalist to get excuses and free ammo. Based off of my last interactions with Remington’s customer service, another important and often lacking part of the production process, I wouldn’t expect much at all. Yes, I am still bitter about my 105 CTi and 105 CTi II. And those two 870s, and a little about that 700 with a cracked bolt head out of the box.

  • Ark

    Design flaws are still design flaws, and for the price tag on that pistol, I expect better quality control. They launched into an incredibly competitive market and utterly failed to follow through.

    • Sunshine_Shooter

      From Remington, one of the biggest firearms manufacturers in the world, quality control should be absolutely unparalleled. What is it? Shoddy, and with no signs of improving soon. Even if they have a good design, the QC ruins it.

      • R H

        Even when they have a great design, they’ll find a way to ruin it. R51 had so much potential. I almost want the big green turd to fail so that someone else can buy them up. Honestly, it couldn’t get much worse,

      • Ark

        I would expect this from Taurus. When you buy a $200 pistol, you expect to do some testing and maybe send it back once or twice before you get a good one.

        When I spend $400 or more, I expect it to freakin’ work.

        • junkman

          No matter what I spend, I expect it to work, without sending it back & with a several hundred round ‘break in’.

      • CommonSense23

        Except that isn’t the business model they are running. They got bought out by a company that just wanted to maximize profits. Quality control cuts into profits. And they got a name that outside of the relatively small gun community garners a lot of respect for the average firearms consumer.

  • t_reese

    I stopped buying anything Remington years ago and will continue to not buy their products till I leave this rock.

  • randomswede

    Lets not forget that the issues with a sample-size-of-one stretches both ways; it’s equally flawed to assume that if a single sample works flawlessly for a reviewer every other sample would too.

    • R H

      But….but….but muh Taurus works fine for the HUNDREDS of rounds I put through it at the indoor range!!!! So they’re all great! UR just mad you bought that overpriced (insert no crap pistol here)!!! Choice-Supportive Bias, what’s that?

      • Button Gwinnett

        Funny you should say that…My Taurus, that I just bought new, ran fine for 275 rounds today. And as I was casing it at the end of the session, the rear sight fell off.

        • CavScout

          If it was a Kel-Tec, the issue with the rear sight falling off would be due to Limpwristing.

      • plumber576

        I think you mean DOZENS of rounds. Have you ever known a Taurus owner that can afford hundreds of rounds?

    • Edeco

      More flawed I would say; I’m too out of shape to say this in proper terms, but one good one doesn’t prove much. The predictive power of a bad sample is greater. I believe it’s what Joseph Juran called the Turd in the Punchbowl Principle.*

      *no, not really.

  • Joe

    Ambi slide stop only works for right hand use.
    Larger/heavier than most comparable handguns.
    Finicky with some ball ammo.
    What superior characteristics of this handgun would cause any potential user to overlook these flaws?

    • Zachary marrs

      Because “its a remington”

  • Paul White

    And this is exactly why I care about customer service; in an ideal world I’d never need it but *lemons happen*, even from good companies like Ruger. Knowing a company will treat me fairly if I do get one really helps.

  • Jeff

    The right side ‘slide catch’ won’t release the slide, I tried one in the store. When something that simple is done wrong, it’s a pretty quick turn off for me. BUT, at least it’s cheap at $399.

  • MrBrassporkchop

    I’d extend this to forum reviews. Even with a larger sample size there’s other stuff going on. Like of there’s a known small issue, every person that had that issue reports it and none of the people that didn’t and are happy with it don’t say anything. Even if the percentage of people unhappy was less than 1%, if they sold a ton of units and a large portion of that less than 1% reports in, it would sound like a monumental failure. Confirmation bias I think? Anyway this is really bad in the gun/tactical industry. Also car industry.

    • USMC03Vet

      It’s pretty telling that faceless randoms in forums have more credibility often than known faces of industries when it comes to product reviews.

      • MrBrassporkchop

        It’s all about their number of posts. Guys with 30k posts under their belt can say anything and people eat it up.

  • USMC03Vet

    YouTube is for entertainment.

    Bring forth the next James “Hollywood” Reeves video, TFB TV!

  • Dougscamo

    Moral of the story is….don’t be the first kind on the block to buy ANYTHING….until it’s out long enough for the manufacturer to work the bugs out….

  • Bullphrog855

    MAC “Torture test” has picked up way to much steam. They were novelty videos, don’t get me wrong, but they are nothing more than that.

    • VanDiemensLand

      It’s not a torture test, It is simply shooting the pistol at the range. Watch the video.

  • Hoplopfheil

    Judging by the follow up, it’s a decent pistol.

    But if I wanted an absurdly chunky 9mm, I’d buy (another) Ruger P95. 🙂

    Maybe it’s just me, but I’m baffled that big bloated new designs are being released. I must not be the market for them. Is it a competition thing?

    • Sulaco

      Police and military market?

      • Hoplopfheil

        Really? For something that fat and heavy? A GLOCK makes more sense for military OR police.

        • Sulaco

          True but if they got a good large purchase deal with Remington they might buy some. Uniform holster carry would make the large and fat problems not matter so much. One never knows what will attract a dept buyer as apposed to a user.

  • I think Remington made a big mistake by using Para mags for the RP9. That nosediving malfunction wouldn’t have happened with a proper magazine pattern.

    • Zachary marrs

      Remington made a mistake?

      Remington IS a mistake

    • Military Arms Channel

      I agree the Para mag usage was probably not the best idea. That, and the feed lips obviously don’t properly retain the rounds in the magazine.

  • ozzallos .

    Sample size of one… and the obsolesce of leverguns comes to mind.

  • That’s the best way to decide on a purchase of just about anything. I know emotion figures into gun reviews many times so the most objective review will be your best bet as well as comparing as many sources as you can. At times we’ll test more than one of the same gun.

  • tony

    my personal approach: wait at least 5 years before buying any new model

  • john huscio

    Yeah that’s nice, but Big Green has an established track record of churning out crap. I expect Remington to fail everytime they release a new product and voila! Failure inevitably confirmed.

    • Robert

      EXACTLY! I any manufacturer would just own up to the screw up, Say “Sorry John Q. We will make it right.” They would develop and keep a loyal following and keep themselves out of $$$ court. Mistakes can be forgiven and rectified. Mistakes that were knowingly sent out for sale anyway… unforgivable! Corporate Ford, Remington, Toyota and the like will never learn this lesson however.

  • Mr. Privilege

    When I want firearms reviews, I only trust NutNFancy. I love spending 90 minutes of my day listening to a guy drone on in an annoying nasal tone doing a “full paleo-review” on a rock and a sharped stick.

  • Tom

    While a sample size of one certainly has it’s issues, I’d bet they pale in comparison to the overly positive experiences that are created when a company hand picks the product in question for review. I’d be willing to bet none of them just pick one randomly off the assembly line and ship it. They are picked, and gone over with a fine tooth comb first.

    You also have the problem of companies giving products for review and allowing reviewers to keep them, because that’s simply just paying for a positive review. People doing that frequently know that if they consistently give less than positive reviews, they are going to stop getting free stuff. The same is true for companies in big magazines that buy ads, if that companies magazine review isn’t favorable there’s the risk of losing ad dollars.

    There used to be a couple cycling and motorcycle magazines that did not take manufacturer based ads and what do you know their reviews were almost always less positive compared to the big $ rags getting lots of ad dollars from manufacturers. Yet when it came to user experiences they were much more closely aligned with the less positive reviews that didn’t rely on ad dollars from the companies whose products they were reviewing.

  • Squatch

    Saw a RP9 at a gun show last weekend. After seeing MAC’s video the first thing I did was lock the slide back and attempt to use the right side release. Can confirm that is does not function what so ever. Other issues aside, this is a massive oversight on Remington’s part to miss such an obvious flaw.

  • MP

    The mag design issue would be all the RP9 mags. The flimsy slide stop would be all RP9 slide stops. That reflects issues with the entire line. Also, this is a sample of two.

  • Charles Valenzuela

    Yeah. When I can see a needle in a haystack as I am approaching the haystack, I have to ask myself, “Am I seeing the only needle in the haystack? How big a coincidence is that?” Ten thousand cans on the shelf. I select one and open it. Instead of green beans it contains half a rat. . . . Am I EVER going to consider buying from that brand again? After the R51, I wouldn’t bother to EVER consider a Remington pistol. Even as a gift.

  • Bob

    I remember years ago someone did a review on a 9mm Makarov surplus pistol and the sample they got was apparently a lemon.
    I bought one that was Hungarian (I’m told) and it works fine and has always worked fine.
    Good pistol. Only complaints I have is that the sights are SMALL for my old eyes (just like a real Colt Gov’t Model 1911)

  • CavScout

    The slide stop flexes like a noodle. That’s not a sample size of one. It also shows a possible lack of testing and a failure of design. At worst.
    It does show that it’ll need more hard testing to figure out what’s weak, etc. Needs a track record first.

  • CavScout

    On the new Rem pistol, I’m almost certain all the issues he had are MAGAZINE RELATED. They need to redesign the mag. This time, make it actually for 9mm and 9mm only.

  • Kalash of the Titans

    I just think it’s funny that Tim has done plenty of other “first shots” videos with quality pistols like CZ, Glock, S&W and has never needed a sample size of more than one. You know the old saying, if it looks like it and smells like it, well….