We always question everything, right? Right. But what if we're faced with a chart, or a graph, or a table of raw numbers? It's easy to fetishize the data, and assume that as long as we have data our arguments must be sound. But as I've talked about before, the data can lie - and more twistedly, the presentation of the data can lie even more. So here are some handy-dandy questions to ask yourself when confronted with data:
- Is the presentation of the data hiding something? Was anything excluded or minimized? Was anything glossed over? Was one part of the graph highlighted or emphasized to draw attention away from something else?
- How were the uncertainties calculated? You can't just pick error bars out of a hat - there's a methodology behind it, and that technique must be explained.
- What are the limitations of the data? What were they not able to cover? Does this introduce a bias?
- What choices were made to lead to the result? Are they explained? Are those choices reasonable or unreasonable?
- What choices were made in the presentation of the data? Percentages or absolutes? Linear or logarithmic? Broken axes? Is there a curve "fit" to the points? Are the data grouped?
- What are some other interpretations of the data?
This is, of course, far from a complete list, and each of these questions could be a note of its own. But it's a good start...