Empirical agreement in model validation

Résumé En

Empirical agreement is often used as an important criterion in model validation (Oberkampf and Trucano 2002; Oberkampf, Trucano and Hirsch 2002; Trucano et al. 2005). However, it is by no means a sufficient criterion as a model can be so adjusted as to fit available data even though it is based on hypotheses whose plausibility is known to be questionable. Our aim in this paper is to investigate into the uses of empirical agreement within the process of model validation as it is performed in scientific practice. In order to do so, we first present the main reason why empirical agreement is not a sufficient criterion for model validation, namely, Duhem problem of refutation and confirmation holism. What we here call "Duhem problem" is the model-oriented version of the Duhem-Quine thesis (Lenhard and Winsberg 2010, Winsberg 2010): When a model's outputs are not the expected ones, the modeler has usually no way to cut the model into pieces that could be confirmed or refutated isolatedly. As a result, she cannot identify which part is responsible for the failure. Conversely, when a model's outputs do agree with available empirical data, it is not easy to tell whether it is only due to adjustments or to the model's core hypotheses. The model faces the court of experimental data as a whole in such a way that it is not easy to determine the precise role of each of its components. However, models do not all suffer in the same way from the Duhem problem. According to their goal and component hypotheses, it is more or less easy to overcome the Duhem problem. Accordingly, empirical agreement is endowed with different meanings in different modeling situations. In order to account for these differences, we put forward a typology of models in the following. At last, we put forward a special type of models that illustrates another difficulty in interpreting empirical agreement. This new difficulty is perhaps even more troublesome for the use of models than is the Duhem problem. 1. The Duhem problem Even though empirical agreement does play an important role in the activity of model validation, it cannot be considered a straightforward criterion for model validation. Here, we understand validity is a purpose-relative notion. A valid model is one that performs the task for which it has been designed, whether predictions, experiment planing, prototype construction, etc. Even though validity is so construed as to be purpose-relative, empirical agreement seems to play an important role it assessing it. Why isn't empirical agreement a simple criterion for model validity, though? Because when a model's outputs are consistent with data acquired by observation or measurement, it is usually not possible to assess to which element within the model this match is due. More precisely, it is not possible to tell whether it is due to adjustments in the model or to the fact that the model's hypotheses accurately represent the underlying processes accounting for the investigated phenomena. Adjustments are mainly of two sorts: a model is "adapted" to the phenomenon at hand either by calibrating it or by introducing ad hoc terms into it. Calibration is a (usually long) process consisting in tuning some parameters-i.e., numerical constants in the model-in order to progressively guarantee that the model 1