how to detect fake statistics

This Spring Quarter I took statistics class in my MBA. The course was mostly about how to use statistics in business application. In the start of the course I was not sure what exactly I will get from it (as it was a mandatory course so I had no choice)- but as I started to dig more in to it- I found that statistics is a very powerful tool that you use throughout your life if you are associated with  business management or business development or politics.

Here is some of the thing I learned on how to spot a fake statistics presented to you on TV, newspaper or journal.

1. Find out who paid for the survey

Take a close look at who paid for the survey. If you read a statistic stating 90% of people lost 20 pounds in a month on a certain “miracle” diet, look at who paid for that survey. If it was the company who owns that “miracle” product, then it’s likely you have what’s called a self-selection study. In a self-selection study, someone stands to gain financially from the results of a trial or survey. You may have seen those soda ads where “90% of people prefer the taste of product X.” But if the manufacturer of product X paid for that survey, you probably can’t trust the results.

2. Look in to sample size- is sample size too small or limited?

Make sure the sample size isn’t too limited in scope. It’s unlikely you can make generalizations about student achievement in the U.S. by studying a single inner city school in Sunnyvale.

3. Look for faulty assumption

For example, you might read a statistic that states unemployment causes an increase in corn production because corn products (like high fructose corn syrup) are cheap and therefore people are more likely to buy cheap foods when unemployed. But there may be many other factors causing an increase in production including an increase in government subsidies for corn. Just because one factor is seemingly connected to another (correlation), that doesn’t necessarily imply causation (that one caused the other).

4. Who was surveyed?

Take a look at if the statistics came from a voluntary survey. A voluntary response sample is a sample where the participants can choose to be included in the sample or not

5.  Beware of precise number or percentage.

If a national survey reports that 3,150,023 households in the U.S. are dog owners, you might be inclined to believe that exact figure. However, it’s highly unlikely (and almost impossible) that anyone would have seriously surveyed all of the households in the U.S. It’s much more likely they surveyed a sample and that 3,150,023 is an estimate and should have been reported as 3 million to avoid being misleading.