The first part of combating data manipulation is recognizing when a graph or chart or data set is misleading in some way.
(1) To understand the first way this can happen, we need to understand “what is measurable”? Something that is measurable has a unit (e.g. length can be measured in inches, weight in grams, etc.). Many advertisements and commercials make use of numbers and measurements that functionally mean nothing, because they are numbers with no units.
So Pantene promises two times more shine in hair; how is shine measured? What does that mean? This number “two times” does not mean anything, but it sounds much more persuasive than saying “you’ll get some more shine in your hair.”
Here is another example. Maybelline promises a 65% lift to the eyelashes, but how do we measure lift in eyelashes? This is persuasive, but definitely not meaningful.
These numbers with no meaning are purely used for persuasion and the first, and probably the easiest, use of numerical manipulation that we can spot.
(2) The second way people manipulate data to persuade is known as “cherry-picking.” This is when certain data is presented, but other data that does not support the point is ignored. Charles Seife in Proofiness presents an example in George Bush’s descriptions of the apparent success of the No Child Left Behind Policy. He claimed that students were in general improving their test scores, ignoring data from 12th grade students whose test scores actually decreased. Secondly, those scores that did increase have actually been increasing slowly for decades, way before the passage of the legislation. Bush cherry-picked data that supported his point and his policy.
(3) The third way is comparing data that should not be compared (e.g. comparing data from different years when the conditions were very different). Seife uses the example of the Blue Dog Coalition criticizing the Bush Administration for borrowing a lot of money from foreign nations. They claimed, “Throughout the first 224 years (1776-2000) of our nation’s history, 42 U.S. presidents borrowed a combined $1.01 trillion from foreign governments and financial institutions according to the U.S. Treasury Department. In the past four years alone (2001-2005), the Bush Administration has borrowed a staggering $1.05 trillion.” While the comparison of numbers sounds incredibly convincing, if you think about it, the value of the dollar in 1776 was vastly different than the value of the dollar today or in 2005, and therefore this comparison does not make a whole lot of sense. This is a common problem with the comparison of test scores from different years; Michael Winerip in The New York Times describes the mayor of New York boasting about the increase in test scores for fourth graders, when many of the educators actually admit that the test just got easier. Therefore, the comparison across years is inaccurate.
(4) Finally, the last type of data manipulation is just changing the graph itself. If one axis of the graph represents only 0-1 (of whatever unit the graph is displaying), it would look vastly different than if the axis went from 0-10 or 0-50. Making sure that the axes and type of graph accurately display the data is extremely important.
Up Next: an in-depth discussion of error (especially as it relates to polls and voting)!
Charles Seife, Proofiness