An example of observer inconsistency

Another way of looking at unique and common noise is by considering this matrix of observer ratings. This data is my own from a recent experiment. Over replications, I responded differently to the same stimulus. This variability is the unique noise. The average rating removes the majority of this unique noise, what is left is the common noise component. The common noise can also be viewed as variability across stimuli for the same event.

This is a very simplistic demonstration of Group Operating Characteristic (GOC) analysis. GOC is a method for removing unique noise by averaging the ratings for the same stimulus. The nature of GOC has been dealt with in a number of Departmental Seminars in the past, by both Vit and John. Although, by definition, the common noise is constant for a particular experiment; the interpretation or inferences that can be drawn from a given experiment will depend on where the common noise came from and how much common noise sampling variability there is.

Common noise sampling variability occurs when an experimenter uses a stimulus set that has been sampled from a population of possible stimuli. It becomes a problem when the experimenter tries to make inferences about the population. However, before I continue describing this I need to give you some background into modern psychophysics, which is based on the theory of signal detectability.