Mike:
For the third time: "If you want reliable information, go to the source........Think carefully before passing on information that you can't trace back to roots in technical literature."
When a technical person answers a question by giving a reference, it does not mean "This is the answer and here is my proof." What they are saying is: "you should go read this article because it discusses the question you are raising and you will either find an answer there or find a set of references you can chase down for more detail." Partially this is a question of conservation of time and space -- not repeating what is elsewhere. Partially it is just how people work. Thus the importance of references and references to references etc. etc.
So the point is not that the peer-review literature is a final authority. The point is to be able to construct an assessment of a line of argument, and a deeper understanding of the subject, by tracing to its technical roots. That means the underlying models, equations, data, graphs, discussion and interpretation of all these, whatever is appropriate. I am proposing you go through the same exercise any technical professional would engage in if attempting to assess the credibility of an author or an argument. I apologize if this was not clear up front. But if you are not able or willing to do that --- and let's be honest, most people aren't, even if you are -- at least use secondary sources whose credibility could, in theory, be checked through that process, by someone else. And if you are not an expert, reference to multiple sources is pretty important to get a balanced and complete perspective.
The reason to use the peer-review literature as a baseline reference is that only there can scientific arguments be assessed at such a root level, at the level of experiment, methodology, analysis, and raw data, for correctness. Sources like WUWT, RealClimate, even AR4, are secondary sources. AR4, and, to some extent, RealClimate, can be fairly directly traced back to peer-review sources. Most web sources cannot. You're welcome to provide counter-examples.
Outside the archival peer reviewed literature, it is not really possible to even say what the complete line of thought, as supported by data, even *is*. (See 2,3,4,6 below.)
Statements about a set of stolen emails, the motivations of the people who wrote them, or what might be read into private confersations of a very small subset of the scientific community, particularly taken from their original context, and, in some cases, altered to change the original meaning, don't have any technical content that can be assessed one way or the other. Such ad-hominem statements are irrelevant to any serious inquiry.
Off the top of my head, here are some reasons why the (archival) peer-review literature needs to be the reference point for discussion.
1. The paper has passed some minimal level of scrutiny with respect to novelty of results and technical quality.
Emphasis on the word "minimal."
This is the most mis-understood aspect of the process and the one that laypeople usually set up straw-man arguments about when they want to duck a scientific discussion.
In theory, the reviewers are not going to let a paper through that contains errors, that doesn't measure up in areas 4,5,6, below, and that doesn't provide a new contribution to knowledge in the field.
In practice, standards vary widely from journal to journal, and editor to editor, reviewer to reviewer. Some journals are very competitive and have very high standards, to the point of snootiness, and others will let through about anything that isn't rank nonsense. Nevertheless, the chances of a paper that has been through this process has of being credible, are, on average, higher than one that has not. Likewise there is, on average, a distinction in quality between what gets published and what not.
2. Archival means a common, fixed reference point for line of argument.
The argument and summary data is preserved in a common form that can be referenced and dissected by anyone, so we all should, in theory, be discussing the same thing when discussing a specific paper. Reduces goalpost-moving. This puts a burden on the authors to either stand behind a paper, or lose credibility. Not true for a web site.
3. It is tracked and indexed.
We can count who cites the paper, how often it is cited, who else is following up, how they use the paper, and if there are any rebuttals or criticisms. It gives a rough guide for people outside the field to what works (and authors) are influential.
4. Papers are expected to contain a detailed description of data and methodology. This, and #5 below, are important points for this discussion. The point at which one can assess the raw data and methodology, and reproduce the results, is the point at which a reference to the literature ceases to become an argument-from-authority.
5. There is an expectation of reproducibility.
6. There is an expectation of summarizing and citing previous work and base sources.
Base material for which there is not room in the publication at hand to discuss should be easily accessible to the reader. This is needed for assembling the complete line of argument, assessing the current data, and performing and reproduction of results desired.
7. Almost always, papers improve with criticism.
In practice, this is probably the most important function of the peer-review process. See #1.
8. Most review processes are conducted blind (authors do not know reviewer's identities) and sometimes double-blind (reviewers don't know author's identities either). This encourages critical commentary by lessening (not removing) fears of retribution and personal feuds.
9. It is an intrinsically adversarial process. Grants, tenure, promotions, and awards are given on the basis of _relative_ status of publication record -- number of publications, how often they are cited, and other impact factors. This provides every incentive to be as critical as possible of competitors work. The closer the work is related, the more likely a review is going to turn nasty, particularly if it looks like the manuscript authors might publish something related to the reviewer's work before the reviewer can.
10. It provides a crude sorting/ranking mechanism.
Something published in Science or Physical Review Letters has probably undergone much more stringent review and selection criteria, and had wider readership and more aggressive follow-up inquiry, than an article in the East Tunisian Journal of Cybernetics, Plastics, and Turnip Growth Studies.
11. Authors can respond to criticisms in an open and hopefully somewhat impartial forum -- often the same forum in which the criticism appears. Built-in mechanism for arbitration of disputes by uninvolved third parties.
In the real push-and-shove world of technical publication, there is a lot of gray area, and at the end of the day, publishing a given paper, or not, is a judgment call on the part of the responsible editor. There are few if any 'perfect' papers -- novel insight, clear and convincing conclusion, solid and complete data, unambiguous interpretation of data. In fact I can't recall having ever read such a paper! Knowledge is built up bit by bit, building a skyscraper out of grains of sand.
At some point, you are correct, an 'argument from authority' takes place. There is no practical alternative. We can't chase every reference and reproduce every experiment. Nobody is an expert in every area. Aside from a few places in pure mathematics, everyone -- and I really mean everyone -- has to rely, at some point, on analysis, interpretation, and data gathering by others. There is, however, a critical distinction between blind trust in some expert's random opinion, and a carefully constructed argument that can, and hopefully has, been laid out for extended criticism and discussion by others. There is also a critical distinction between a selective but careful inquiry in primary literature, and ignoring that literature altogether.
As far as climate science being a 'an immature science dominated by a relatively small group of researchers' -- this also can be checked. Recently I pulled the reference list from AR4, Ch.8. 685 references in that chapter alone. You can go count how many unique researchers there are. The first chapters of the climate text I mentioned heavily reference standard texts in classical subjects. Even by the mid-1980s the field was mature enough to warrant wide-scope reviews as in IPCC AR1.
Summarizing:
The peer-review literature is not the "gold standard" for knowledge. It is the repository for raw knowledge and the place where serious scientific conversations take place. If a technical argument is not in that literature, in some form, there is no realistic way of assessing, developing, criticising, explaining, or futhering it. It also happens to be the only system going so there's not really a practical alternative base for discussion.
If you think there are flaws in that literature, the solution is -- write the paper explaining the flaws and submit it for publication. If someone on one of the secondary -- but heavily referenced -- sources I listed initially is mis-stating what is in the literature, then that should be pointed out, they need to be corrected.
Otherwise, what is your point? That we should dispense with all the items 1-11 above, ignore expert analysis and data, and make decisions based on what we read on our favorite web sites?
Reply With Quote
Bookmarks