Temperature data "adjustments" off limits

[Little Fish]

"Prof Jones has already accepted he should have been more open, and has since made all the station data referred to in these emails publicly available," Dunford told FoxNews.com.

Watts said that while much of the data itself is now available, the methods of adjusting it -- statistical modification meant to filter anomalies, "normalize" the data, and potentially highlight certain trends -- remain a secret.

"Much of climate science, in terms of the computer processing that goes on, remains a black box to the outside world. We see the data go in, and we see the data that come out as a finished product -- but we don’t know how they adjust it in between.”

Watts said he would like to be given the adjustment formulas to make his own determination.

"The fact that they are trying to keep people from replicating their studies -- that's the issue," Watts noted. "Replication is the most important tenet of science."

http://www.foxnews.com/scitech/2011/12/16/complicit-in-climategate-doe-under-fire/

[Von Bismarck]

"Prof Jones has already accepted he should have been more open, and has since made all the station data referred to in these emails publicly available," Dunford told FoxNews.com.

Watts said that while much of the data itself is now available, the methods of adjusting it -- statistical modification meant to filter anomalies, "normalize" the data, and potentially highlight certain trends -- remain a secret.

"Much of climate science, in terms of the computer processing that goes on, remains a black box to the outside world. We see the data go in, and we see the data that come out as a finished product -- but we don’t know how they adjust it in between.”

Watts said he would like to be given the adjustment formulas to make his own determination.

"The fact that they are trying to keep people from replicating their studies -- that's the issue," Watts noted. "Replication is the most important tenet of science."

http://www.foxnews.com/scitech/2011/12/16/complicit-in-climategate-doe-under-fire/

Wasn't Richard Muller allowed?

[Doug1029]

Watts said he would like to be given the adjustment formulas to make his own determination.

You might send Watts this:

Data is normalized with this formula:

Y' = (y(i) - ybar)/s

Where: y(i) is the value of y before normalization,

ybar is the average value of y,

s is the standard error of y, and

Y' is the new value of y.

It's in practically every statistics book that was ever written. Mr. Watts needs to do some reading.

Doug

[Guest]

Still referencing Anthony Watts as if he's a reliable source

Consistency at least

Br Cornelius

[Doug1029]

"Prof Jones has already accepted he should have been more open, and has since made all the station data referred to in these emails publicly available," Dunford told FoxNews.com.

Watts said that while much of the data itself is now available, the methods of adjusting it -- statistical modification meant to filter anomalies, "normalize" the data, and potentially highlight certain trends -- remain a secret.

"Much of climate science, in terms of the computer processing that goes on, remains a black box to the outside world. We see the data go in, and we see the data that come out as a finished product -- but we don’t know how they adjust it in between.”

Watts said he would like to be given the adjustment formulas to make his own determination.

"The fact that they are trying to keep people from replicating their studies -- that's the issue," Watts noted. "Replication is the most important tenet of science."

http://www.foxnews.com/scitech/2011/12/16/complicit-in-climategate-doe-under-fire/

Watts' comments say more about the integrity of his reporting and his ability to read technical papers than they do about conspiracies.

Scientific papers are not written for the likes of Watts. They are written for other scientists, which means the reader has to have a pretty good understanding of the language, concepts and processes involved, or the paper will make no sense.

One can download the program ARSTAN from the NCDC dendrochronology site for free. It is also available from Henri Grossino-mayer's Ultimate Tree Ring website, or PM me and I'll email you a copy. The first page to appear on the screen lists about a dozen models/methods commonly used in dendrochronology.

The way in which the various equations are used is at least as important as the equations themselves. If Mr. Watts does not use them in the right way, he will be unable to reproduce the results.

The first step in the analysis is to remove variability correlated with tree age ("detrending"). This is done with a combination of "models" (processes based on an equation) or "methods" (processes not based on an equation). These models and processes have names like "Hugershoff's Model," "seven-year cubic spline," "negative logarythm" and "running average." Practicioners know (or can easily find) the equations; Mr. Watts is reading these names without realizing that they are the equations he is complaining about not having.

The researcher chooses the detrending models/methods he wants to use based on what he is trying to do and what the particular model/method is good at. Hugershoff's model works well with old trees, but not so well with young ones; the negative logarythm is better for young trees. These are arranged in a "hierarchy" in the order in which the program is to use them. A commonly-used hierarchy is (1) Hugershoff's model, (2) negative logathrym, (3) straight line and (4) none.

In the above hierarchy, the program will first fit Hugershoff's model to the ring series of a given tree. This is a non-linear model and worse, requires successive iterations to converge on a solution. Computer programs solve this equation by the "brute force" method: trying different values for each coefficient until they find the best-fit combination. If ALL resulting coefficients are significant at the 95% confidence level, the program accepts Hugershoff's model as the best fit and skips the rest.

If Hugershoff's model fails, then the program proceeds to the negative logarythm. This is a nonlinear model, but the easiest way to fit it is to run the data through a "transform" first. In this case, the program fits a straight-line model to the logarythms of the ring-width measurements. It again applies the 95% confidence test to the coefficients. In practice, natural logs are usually used.

If the negative logarythm fails, the program fits a straight line model to the series. This is computed directly from the raw data without transforms and the like. It again applies the 95% test.

If the straight line fails, the data is used without being detrended.

Once a model is chosen, the program uses the model to estimate a value for each observation, subtracting that from the measured value to leave a "residual." The average value of the measured ring widths is added to each residual to give a series of ring-widths that are not affected by age.

This process may remove other variability from the series (like the long-term climate signal, for example). Also, it may not remove all age-related variability. To solve the latter problem, a technique called "krigging" may be used. In this method, each successive model in the hierarchy is applied to the residuals from the previous model. At the end of the process, either most age-related variability has been removed, or it wasn't there to begin with.

The next step is to remove variability associated with "autocorrelation" (the tendency of adjacent rings to be of similar size). The process is similar to the technique above, so I won't go over it here.

Once all this is completed, one has a series for one tree that has no age-realted or autocorrelated variability. To obtain a climate signal that is free of this variability, one takes the original observation and subtracts the corresponding estimate, adding the average back in.

Now repeat the process for each tree in the dataset. That should keep Mr. Watts busy for a long time.

To say that there is some kind of conspiracy to keep the equations and models secret is absurd when they are published and available to anyone who wants them. Mr. Watts is not being truthful with his readers.

If Mr. Watts, or anybody else, wants the actual equations, I will be glad to furnish them.

Doug