Though by this point old news, the 2007 Aberdeen rifle dust tests (link starts a download) have caused a lot of consternation on the Internet since their publication. Championed by some as final proof of the “defective” design of the AR-15, and criticized by others as being imprecise, unrepeatable, and almost meaningless, they still have, in this author’s mind, considerably colored the rifle reliability discussion to this day.
Not long after the release of the test results, SMSgt Mac, a reliability and maintenance technician who runs the Elements of Power blog, wrote a critique of the tests, going above and beyond the call of duty to explain to us laypeople (albeit, perhaps those laypersons who have taken a statistics class) what the shortcomings of the tests were, and why they contribute so little to the discussion:

Using Slide 13 for perspective, we can view this data and say that IF the Extreme Dust Test data is valid and representative of the real world (and we have every reason to believe that the real world is a more benign environment) then the largest average disparity we might find in C1 & C2 stoppages between any two weapons for an engagement that consumed one basic load would be less than 1 stoppage difference for every TWO engagements. If for some reason the soldiers started shooting 2 basic loads on average, the greatest average difference in numbers of stoppages between different weapon types for one engagement would be about 1 ½ stoppages per engagement. Because of the absence of detailed failure data by specific weapon, failure, and failure mode we cannot determine whether or not this information is ‘good’ or ‘bad’ — even if this data was representative of the ‘real world’. If for instance the M4 (or as has been noted possibly the HK416) had one ‘bad actor’ in the bunch, it would have completely skewed the test results. If we cannot even tell if THIS difference is significant, we STILL cannot assert any one weapon is ‘better’ than another even within the confines of this test. All we still KNOW is that the M4 experienced more failures. The good news is, the Army will have a better idea as to what they need to do to perform a better test the next time.

…

The only thing I fear coming out of these test results is that out of the emotion behind the concern, perhaps this test’s importance will be blown out of proportion within the total context of what a Soldier needs a weapon to do. I can see us very easily buying the best-darn-dust-proof-rifle-that-ever-t’was… and then spend the next twenty years worried about it corroding in a jungle someplace.

Despite the post’s age, I think some TFB readers may find it interesting. The dust tests appear to have since served their intended purpose, and the Army has moved on. What is made clear here is that the understanding of data is just as important as the data itself. With a flawed understanding, or worse, a deliberately twisted interpretation of the data, the potential knowledge to be gained through experimentation is at risk.