How useful is the ideal observer? We can't run an experiment to test it directly because there is no way of presenting a real observer with all the signals in the signal set. As is typical, we take a sample of waveforms from the signal set of interest. We then infer or generalise the result back to the population.

But what is the theoretical model now? Is it exactly the same as the mathematical model that assumed all waveforms from the signal set.

No it isn't.

What results in fact is an infinite number of mathematical models. A different model for each random sample and for each sample size. This results in an infinite number of ROC curves. I will call these types of ROC curves SAMPLE ROC curves and the corresponding mathematical model the sample model.

Now this wouldn't necessarily be a problem if we knew what the theoretical model was in a real detection task e.g., a human detecting band limited white Gaussian noise. We could, in many cases, re-derive the ideal observer using only the particular signals in the sample set instead of assuming all possible signals.

More often than not, however, we do not know what the ideal observer or mathematical model is. Therefore we also don't know what the sample model is. We can run the experiment with the sample but we will have problems interpreting the results unless we have some idea how the sample model relates to the theoretical model. Only then can we make a judgment as to whether our data is consistent with the theory.

What are the characteristics of the sample models and sample ROC curves? How can we assess how close the sample model is to the true theoretical?