First page
Back Continue Last page Summary Graphics
Why is common noise sampling variability a problem?
If the sample ROC curve is extreme compared with the true theoretical ROC curve then we won't know if our experimental results are consistent with the theory.
Comparing two similar theoretical models may also be impossible because the sample ROC curve could possibly come from either. Replicating an experiment with a different sample set may result in quite different results. The interpretation of this is difficult, it could imply that the phenomenon could not be repeated or it may be just two extreme sample ROC curves.
The aim must be to minimise the common noise sampling variability. One obvious way of minimising it is by using more samples. There is a trade-off involved, however, because you can’t just add thousands of stimuli - there are practical limits to how long a replication should take.
If we determine the function relating variability of the area to number of samples we should be able to determine the minimum number of samples required, based on the variability we are willing to tolerate.
Now in most cases of interest we don't know what the theoretical distributions are. However, in many experiments there is some theoretical model that is being tested. As a starting point we can use the model to predict the amount of variability as a function of samples taken from the evidence. This can be estimated by simulating the sampling signal+noise and noise alone distributions and looking at the resultant sample ROC curves and area.
If we repeatedly take samples of the same size we can calculate the mean and standard deviation of the sample ROC area.