12.13: Exercise- Using Area Under the Curve (AUC) with the ERP CORE P3b Oddball Paradigm

In this exercise, you’ll take what you learned about decoding with different numbers of trials per class and apply this knowledge to classifying rare targets and frequent standards in the data from the ERP CORE P3b paradigm. The paradigm was described in detail in Chapter 6. To keep things simple, we’ll do the decoding only for Subjects 1–5. The preprocessed EEGsets for all 34 subjects can be found in the P3b_Data folder. The necessary preprocessing steps have already been applied (e.g., filtering, referencing, artifact correction, bin assignment, epoching). Bin 1 consists of the Rare Targets, and Bin 2 consists of the Frequent Standards. I’ve excluded trials with behavioral errors (because the P3b depends on how the subject actually responds), but I haven’t excluded trials with artifacts (because, as described earlier, this tends to hurt decoding more than it helps).

I’ve created conventional ERPs for Subjects 1-5 (in the P3b_Data folder), and Screenshot 12.19 shows the grand average of these 5 participants at the Pz electrode site (where the P3b was largest). You can see that the P3b component was much larger for the Rare Targets than for the Frequent Standards.

We can potentially gain sensitivity and statistical power by using decoding rather than conventional methods to quantify the extent to which the ERPs are different for the Rare Targets versus the Frequent Standards. However, we need to do the decoding in a way that is not distorted by the fact that there were approximately four times as many Frequent Standards as Rare Targets. As recommended at the end of the previous section, we're going to use the Equal Trials, Max AVGs approach to equate the number of trials while using the maximum number of averages/pseudotrials per class.

Begin by clearing any loaded BESTsets and MVPCsets from eStudio. I’ve already converted the EEGsets into BESTsets, so go to the BESTsets panel in the Pattern Classification tab, click Load, navigate to the P3b_BEST folder, and load the 5 BESTsets. Select all 5 BESTsets in the BESTsets panel, and then set up the Multivariate Pattern Classification panel as shown in Screenshot 12.20. We have only 30-40 trials in the Rare Targets bin, and we want to have 10-20 trials per average/pseudotrial, so we’re setting Cross-validation Folds to 3. In the bottom portion of the panel, set the Number of trials and averages section to Equal Trials, Max AVGs and Across Classes, and click Show summary of trials and averages. For Subject 1, you should see that we have 11 trials per average and 3 averages for the Rare class, and 11 trials per average and 14 averages for the Frequent class. Thus, we’ve equated the number of trials per average, but we’re using more averages for the Frequent class than for the Rare class so that we don’t have to throw away a bunch of trials from the Frequent class.

Close the summary window and then click on the Options button in the decoding panel. Check the Downsample box and set it to decode every 19.53 ms. The sampling rate for this experiment was 256 Hz, which means that there was one sample every 3.91 ms. This is why we are decoding every 19.53 ms instead of every 20 ms. 

Click Okay to go back to the decoding panel, and make sure that the Metric is set to ACC. Now you can either click Run or load the MVPCsets that I’ve already created in the P3b_MVPC folder. If you plot all five of the MVPCsets, you should see something like Screenshot 12.21A. And if you make a grand average of those MVPCsets, you should see something like Screenshot 12.21B.

Now we’re going to repeat the decoding using area under the curve (AUC) as the metric of decoding performance. You can learn exactly what AUC means and how it’s calculated by reading Chapter 11 of the third edition of An Introduction to the Event-Related Potential Technique (Luck, in preparation). Briefly, AUC captures the ability of the decoder to distinguish between the classes in a manner that controls for any biases toward one or the other class. Chance is always an AUC of 0.5, and perfect decoding would be an AUC of 1.0.

Let’s try it. Keep everything the same as before, except change the Metric to AUC, and then run the decoding (or load the MVPCsets that I’ve already created in the P3b_MVPC_AUC folder). If you plot all five of the MVPCsets, you should see something like Screenshot 12.21C. And if you make a grand average of those MVPCsets, you should see something like Screenshot 12.21D. Note that the Y axis in these plots is now AUC instead of Decoding Accuracy.

The plots of decoding accuracy (ACC) and area under the curve (AUC) look pretty similar. However, the standard errors are a little lower for the AUC metric, indicating that the variability among participants was a little lower. That should mean that the effect size (Cohen’s d_z) should be greater for AUC than for ACC. There isn’t a direct way to compute this with eStudio, so I wrote a little script for it. The results are shown in Figure 12.4. As expected, the effect size was at least a little larger for AUC than for ACC at most points, and it was much larger for AUC than for ACC from approximately 300-500 ms. This is consistent with a study conducted by Guanghui Zhang, in which he compared ACC and AUC across a large number of decoding studies and found that the effect sizes were typically larger for AUC (Zhang et al., in preparation). 

---