A recent thread of discussion (namely, some comments on the hybrid cars podcast episode) brings me to remind you of the wisdom of cross-checking ostensibly “scientific” things you hear in conversation, or see mentioned in print or online somewhere.
Fortunately, thanks to the internet, it’s pretty easy these days to do a little quick sanity checking. When I run across something that smells questionable, I’ve gotten into the habit of looking for references to the fishy material on Google. I call this “googlediving” — it’s essentially the information equivalent of dumpster diving but doesn’t leave you all smelly when you’re done. Best of all, it usually takes just a few minutes.
Let’s look at the recent discussion as a good example. It all started with a link (thanks, darek!)to a newspaper editorial, based on a study purporting to show that the lifecycle energy consumption of a Prius is greater than that of a Hummer. The study was conducted by a company called CNW Marketing Research, and was first released in the summer of 2006, but was updated and re-released in the spring of 2007. Each time, a flurry of commentary (both online, and in dead-tree outlets) ensued.
So let’s go diving, shall we?
A quick Google search on “CNW Prius Hummer energy” reveals a common pattern for the down-stream appearance of controversial press releases like this:
A number of sites accepting the information non-critically (“…gosh, I guess I’m not going to buy that Prius now…,” “see, I told you those darned environmentalists are idiots…,” etc.). Political blogs and commentators tend to fall in this “confirmation bias” group.
A number of sites rejecting the information non-critically (“…they must be shills, paid by General Motors…”).
A handful of critical looks at the study and its source.
Obviously, I’d recommend you focus your attention on the third class of sites.
I have to give the CNW folks some credit — people (myself included) are accustomed to thinking of vehicles more in terms of fuel economy than of lifetime, end-to-end energy usage. It’s not so easy for a layman (layperson?) to determine how much energy is consumed in the production of a vehicle. CNW apparently compiled their survey based on a large number of questionairres, tallying up even the miles driven by workers headed to individual factories. But enthusiasm can’t make up for bad methodology. And in this particular case, I’d have to argue that the the CNW study has two fatal flaws: opacity, and sloppy analysis.
Only a “non-technical” summary of the results of the study was published (458 pages!), while the original data, assumptions used, and study methodology are hidden behind a “proprietary” wall. Also note that this study wasn’t subjected to peer review, and not enough information was released to enable peer review anyway. The survey authors seem quite proud of the fact that their methods and source data are closely-held secrets — I’d argue that this is appropriate if you’re trying to sell newsletter subscriptions, but not at all appropriate for a study attempting to present itself as scientific. Meanwhile, the source of the study is a small operation that focuses on truck & car marketing — this isn’t a killer, but raises questions about their analytical skill set. If nothing else, it goes a long way toward explaining why their study is so different from what a proper scientific publication should look like.
I’d argue that a better approach is one that proceeds from a scientific (vs. market analysis) approach — as, for example, is taken by an MIT group looking at the lifecycle energy “cost” of a variety of transportation options. In the MIT study, all the data & methodology used is openly documented in exhaustive fashion. Oh, and its conclusions are diametrically opposed to what CNW is pushing.
Also, I’d suggest you read Peter Gleick’s analysis of the CNW study, published by the Pacific Institute (although it’s admittedly not an unbiased party). These are just a few particularly interesting items from Gleick’s paper for your consideration:
CNW’s results disagree with those of a number of previous studies, which have shown that the production of a vehicle uses less than 10% of its whole-life energy consumption (vs. CNW’s study, which reports that the vast majority of a vehicle’s life-cycle energy consumption takes place in its construction).
CNW’s assumed vehicle lifetimes have weak (often inconsistent) justifications, and in many cases are contradicted by user experience.
CNW’s energy costs of factory construction are amortized in an inconsistent fashion.
In CNW’s study, similar vehicles of similar construction and fuel economy, built on the same assembly line and distributed through the same network, are given dramatically different lifetime energy consumption values — raising questions about the methodology used in the report’s analyses.
Repeated misuse of simple concepts / units for power vs. energy in CNW’s published report raise additional questions about the soundness of its analyses.
Correcting just a few of the identified flaws in the study would reverse its results.
There’s also a good analysis on Gristmill. Anyway, I’ll leave you to peruse the above material, and draw your own conclusions. Then, please leave a comment!