The subject of this blog post is "What constitutes valid criticism of a scientific model?" (in relation to its predictions). Certainly there seems to be much dubious criticism of scientific models particularly, but not only, in the mainstream media. I'll give some examples in the form of thought experiments first and more concrete examples later.
By model I mean a simulation which is run in an attempt to derive predictions from a theory. The first thing to note about such models, is that they are almost always inexact! That is, the model is not completely accurate in its simulation of the scientific theory in question. The reader might find it illuminating to search for counterexamples.
Models, however, can be more or less accurate, and certain models are closer to the underlying scientific theory. Generally, models that are closer to the underlying scientific theory are better. But as a practical matter we cannot demand that all models are completely exact because this currently requires (and probably always will) more computing power than we have available.
So the inexactness of a model on its own is not a practical way to criticize that model. I give a few principles that I think any valid criticism of a model should obey. Firstly though, I should point out that a model may be accurate in one area and woefully inaccurate elsewhere. So we shall not be talking about criticizing a model itself but criticizing the model's use in making certain predictions.
Suppose that Alice has a model of some scientific theory. Suppose in addition that Ben has found one or more inaccuracies in the way Alice's model simulates this scientific theory and Ben is claiming that this undermines certain predictions made by Alice from the theory with the help of the model. Then my criterion (necessary but not sufficient) for Ben to have a valid criticism, are as follows:
1) The inaccuracy must affect the prediction in question.
Suppose Alice has a model of celestial mechanics that places the earth about one thousand kilometers further from the sun than it actually is. Assume that the model is otherwise reasonably accurate in the size, shape, motion and spin of the earth, sun and moon. This model will then predict the existence of seasonal variation in the apparent intensity of the sun.
If Ben criticizes Alice's model on the basis of its error then he is not making a valid criticism.
2) The inaccuracy must be significant enough to change the prediction in question.
Suppose Alice uses the above model to predict the timing of the tides. Now suppose that Alice's model is used to calculate the tides for a given day, a month in the future, to a claimed accuracy of 1 hour. The model will give a correct prediction, despite this error in the model. The error is simply not significant enough, to make a difference in the calculation (the sun is many millions of kilometer from the earth).
Now suppose in addition, that Ben's criticism is intended to show that an alternative prediction is in fact correct (or more weakly that the correct prediction lies in a certain direction from that originally calculated). Then I give further criterion for Ben's criticism to be valid:
3) A modified model must be given that corrects the given inaccuracy and this model must make a prediction which supports the alternative prediction.
Let Alice have a model of photosynthesis which overestimates the weight of a molecule of carbon dioxide. Suppose Alice's model predicts the efficiency with which some type of plant converts solar energy into sugar. Suppose Ben criticizes Alice's model saying that it overestimates the actual predicted efficiency. It is reasonable to ask Ben for an alternative simulation showing that the known inaccuracy actually affects the prediction in the manner asserted.
4) The relevant prediction of the modified model must itself be robust to criticism on the basis of any inaccuracies the modified model still has.
Consider again, Alice's model of photosynthesis. Suppose that this model, was also inaccurate, in its estimate of the weight of a water molecule. Then suppose Ben, gives an alternative model, that is more accurate in its assumption, regarding the weight of a carbon dioxide molecule. I can get a third model by adjusting Ben's model so that the weight of a water molecule is more accurate. Now suppose my model makes predictions that are closer to those made by Alice's model than those made by Ben's model. Then it is not clear whether Alice's model or Ben's model is closer to calculating the correct predictions as far as the original theory is concerned. There may of course be further inaccuracies present in my model.
The above four points can be summarized as "You should trust the predictions of a model (assuming you trust the underlying theory) if those predictions are robust under various improvements of that model". However, in practice, it may often be easier to identify invalid criticism, by referring to the above four points, in the order in which I've given them.
Of course, this also cuts both ways. When Alice proposes her model, then she should estimate the error in the models calculations, by considering various improvements to her model, and seeing how much these affect the result of calculations based on her model.
In weather forecasting, many simulations are performed in an attempt to estimate the uncertainty in any given prediction. Similar techniques are used to estimate the uncertainty in our global climate models.
Now for some more concrete examples.
A) Global warming is unlikely because climate models are inaccurate in X,Y and Z.
Many assumption underlying climate models may be gross approximations but the global warming skeptic needs to show that it is likely that these inaccuracies actually affect the prediction that releasing CO2 will cause global warming.
I am informed, by those who know more about the climate than myself, that this is not the case.
B) Consider a simplified economic model, that assumes no collusion between companies. Then the price of X, will not be reduced by a decreased cost in the production of X as this model would predict, because companies will collude to prevent this happening.
This depends on what X is! The argument needs to be supported with actual adjusted models, that include some details of the game theory involved in company interactions. The accuracy of the models, is best tested empirically here (checking them directly from quantum mechanics is not an option).
By model I mean a simulation which is run in an attempt to derive predictions from a theory. The first thing to note about such models, is that they are almost always inexact! That is, the model is not completely accurate in its simulation of the scientific theory in question. The reader might find it illuminating to search for counterexamples.
Models, however, can be more or less accurate, and certain models are closer to the underlying scientific theory. Generally, models that are closer to the underlying scientific theory are better. But as a practical matter we cannot demand that all models are completely exact because this currently requires (and probably always will) more computing power than we have available.
So the inexactness of a model on its own is not a practical way to criticize that model. I give a few principles that I think any valid criticism of a model should obey. Firstly though, I should point out that a model may be accurate in one area and woefully inaccurate elsewhere. So we shall not be talking about criticizing a model itself but criticizing the model's use in making certain predictions.
Suppose that Alice has a model of some scientific theory. Suppose in addition that Ben has found one or more inaccuracies in the way Alice's model simulates this scientific theory and Ben is claiming that this undermines certain predictions made by Alice from the theory with the help of the model. Then my criterion (necessary but not sufficient) for Ben to have a valid criticism, are as follows:
1) The inaccuracy must affect the prediction in question.
Suppose Alice has a model of celestial mechanics that places the earth about one thousand kilometers further from the sun than it actually is. Assume that the model is otherwise reasonably accurate in the size, shape, motion and spin of the earth, sun and moon. This model will then predict the existence of seasonal variation in the apparent intensity of the sun.
If Ben criticizes Alice's model on the basis of its error then he is not making a valid criticism.
2) The inaccuracy must be significant enough to change the prediction in question.
Suppose Alice uses the above model to predict the timing of the tides. Now suppose that Alice's model is used to calculate the tides for a given day, a month in the future, to a claimed accuracy of 1 hour. The model will give a correct prediction, despite this error in the model. The error is simply not significant enough, to make a difference in the calculation (the sun is many millions of kilometer from the earth).
Now suppose in addition, that Ben's criticism is intended to show that an alternative prediction is in fact correct (or more weakly that the correct prediction lies in a certain direction from that originally calculated). Then I give further criterion for Ben's criticism to be valid:
3) A modified model must be given that corrects the given inaccuracy and this model must make a prediction which supports the alternative prediction.
Let Alice have a model of photosynthesis which overestimates the weight of a molecule of carbon dioxide. Suppose Alice's model predicts the efficiency with which some type of plant converts solar energy into sugar. Suppose Ben criticizes Alice's model saying that it overestimates the actual predicted efficiency. It is reasonable to ask Ben for an alternative simulation showing that the known inaccuracy actually affects the prediction in the manner asserted.
4) The relevant prediction of the modified model must itself be robust to criticism on the basis of any inaccuracies the modified model still has.
Consider again, Alice's model of photosynthesis. Suppose that this model, was also inaccurate, in its estimate of the weight of a water molecule. Then suppose Ben, gives an alternative model, that is more accurate in its assumption, regarding the weight of a carbon dioxide molecule. I can get a third model by adjusting Ben's model so that the weight of a water molecule is more accurate. Now suppose my model makes predictions that are closer to those made by Alice's model than those made by Ben's model. Then it is not clear whether Alice's model or Ben's model is closer to calculating the correct predictions as far as the original theory is concerned. There may of course be further inaccuracies present in my model.
The above four points can be summarized as "You should trust the predictions of a model (assuming you trust the underlying theory) if those predictions are robust under various improvements of that model". However, in practice, it may often be easier to identify invalid criticism, by referring to the above four points, in the order in which I've given them.
Of course, this also cuts both ways. When Alice proposes her model, then she should estimate the error in the models calculations, by considering various improvements to her model, and seeing how much these affect the result of calculations based on her model.
In weather forecasting, many simulations are performed in an attempt to estimate the uncertainty in any given prediction. Similar techniques are used to estimate the uncertainty in our global climate models.
Now for some more concrete examples.
A) Global warming is unlikely because climate models are inaccurate in X,Y and Z.
Many assumption underlying climate models may be gross approximations but the global warming skeptic needs to show that it is likely that these inaccuracies actually affect the prediction that releasing CO2 will cause global warming.
I am informed, by those who know more about the climate than myself, that this is not the case.
B) Consider a simplified economic model, that assumes no collusion between companies. Then the price of X, will not be reduced by a decreased cost in the production of X as this model would predict, because companies will collude to prevent this happening.
This depends on what X is! The argument needs to be supported with actual adjusted models, that include some details of the game theory involved in company interactions. The accuracy of the models, is best tested empirically here (checking them directly from quantum mechanics is not an option).
Just a quick note to say great to see people blogging about climate change and if you want more information on the topic you might like to check out my blog. Links are always welcome, if you like the site but more importantly it may provide you with a few interesting stories for your readers. Also the morning is not incredibly overstaffed while the evening gets the shaft. This extra labor trends across all day.
Posted by: viagra online | 08 September 2010 at 18:20