Variability of ROC curves as sample size is increased

This overhead shows examples of the decrease in sampling variability in the set of sample ROC curves, as more samples are added. This is for the Rayleigh-Rayleigh ideal observer. This one shows 100 sample ROC curves based on 50 samples per event. This is equivalent to using 100 stimuli in an experiment. It should be reasonably obvious that the possible performance of an observer could be anywhere. Even at 200 samples per event, the variability is still reasonably large. At 1600 samples per event the variability of the set of sample ROC curves is smaller but still obvious. The difference between some psychophysical theories may be within this range. At over 800 thousand samples per event, the variability is virtually nonexistent.

However, there is no way you’d ever present a real observer with this many stimuli - I conservatively estimated that it would take 48 days to run one replication!!!