12.5: Exercise- Decoding Multiple BESTsets and Making a Grand Average

Now that you’ve seen how to decode the data from a single participant, we’re going to decode multiple participants in one shot and then make a grand average. To keep things simple and fast, we’ll just work with the data from subjects 301-305. I will assume that you have already decoded subject 302 and that the MVPCset for this subject is still loaded. Before getting started, click Clear All in the BESTsets panel and also in the MVPCsets panel.

I’ve already made the BESTsets for you. In the BESTsets panel, click Load, navigate to the BESTsets folder (inside the Chapter_12 folder), and open the files named 301.best, 302.best, 303.best, 304.best, and 305.best. Now select all five of these BESTsets in the BESTsets panel and go to the Multivariate Pattern Classification panel. Make sure the parameters are still set as shown in Screenshots 12.1 and 12.3 from the previous exercise.

Click the button near the bottom of the decoding panel labeled Show summary of trials and averages. You will now see information about all five of the BESTsets. One thing you should note is that, whereas there should be 160 trials in each class/bin, subject 301 has 190 trials in Bin 1 and 172 trials in Bin 2. This is the result of a small problem with the stimulus presentation script for this subject. 

Because the trials for each bin are subdivided into 10 sets for making averages/pseudotrials (indicated in the B(ERPs) column), we have enough trials to have more than 16 trials per average in subject 301. As you will see in the N(per ERP) column, which indicates the number of trials per average, the subjects with 160 trials per class have 16 trials per ERP. However, subject 301 has 17 trials per ERP. That makes sense for Bin 2, which has 172 trials, but shouldn’t there be 19 trials per ERP for Bin 1, which has 190 trials?

The reason for this is that it’s very important to have the same number of trials per average for all bins/classes being decoded. As you will see in a later exercise, decoding accuracy tends to be artificially inflated when the number of trials per average differs across bins/classes. When you set up the main decoding panel, you selected the Equal Trials, Equal AVGs and Across Classes options, which told eStudio to subsample from the available trials for each bin/class to equate the number of trials per average. That is why both Bins 1 and 2 have 17 trials per average for subject 301.

Do you also need to equate the number of trials per average across participants? As long as you’ve equated the number of trials per average within each individual participant, you will not artificially inflate the decoding accuracy (i.e., produce decoding accuracy that is greater than it should be given the information in the EEG). Consequently, it is not always necessary to equate the number of trials per average across participants.

However, decoding accuracy is greater when more trials are available for averaging (all else being equal). Thus, you will need to equate the number of trials across participants if you are comparing decoding accuracy between two experimental conditions (e.g., comparing letter decoding accuracy for attended versus unattended stimuli) or between two groups (e.g., comparing letter decoding accuracy for a patient group and a control group) or if you are looking at individual differences (e.g., assessing whether letter decoding accuracy is correlated with scores on a reading test). This can be achieved by loading all the BESTsets into eStudio and checking the box labeled Across BESTsets in the decoding panel. This will equate the number of trials per average for all the loaded BESTsets, subsampling as necessary from the available trials. [Alternatively, you can manually determine the appropriate number of trials per average and select the Manual Floor option to specify this number.]

Let’s try it! Close the window showing the summary, check the box labeled Across BESTsets, and click Show summary of trials and averages. You will now see that all four subjects have 16 trials per average. Now close the window showing the summary and uncheck the box labeled Across BESTsets. It’s not necessary to equate across participants in the present study, because we are not comparing across groups or experimental conditions with different numbers of trials. There’s no point in throwing away trials unnecessarily, and it would just reduce our statistical power in the present study.

Box 12.1 describes some common situations in which the number of trials will vary across groups or conditions, as well as a trick that can sometimes avoid the need to throw away large numbers of trials.

Box 12.1 Why the Number of Trials per Average May Vary (and a Trick)

Sometimes the number of trials per bin/class differs widely because of the experimental design. For example, my lab ran an oddball experiment in which there were 4 stimulus locations that appeared on 87.5% of trials and 4 stimulus locations that appeared on 12.5% of trials (Carrasco et al., 2025). We decoded the location of the stimulus separately in the frequent and rare conditions, and then we compared decoding accuracy for the frequent and rare conditions. To avoid having greater decoding accuracy for the frequent condition simply because more trials were available for decoding, we equated the number of trials used for decoding in the rare and frequent conditions. We found that location decoding accuracy was slightly better for the rare set of locations than for the frequent set of locations, especially after 800 ms poststimulus.

Artifact rejection is another common cause of differences in the number of trials per bin/class. In many cases, this leads to modest random variations in the number of trials per bin/class, in which case the Equal Trials, Equal AVGs and Across Classes options will solve the problem. However, if you are comparing decoding accuracy across experimental conditions, across groups, or across participants, there may be systematic (nonrandom) differences in the number of trials marked for rejection. For example, blinks might be more common in a more difficult experimental condition, in a cognitively impaired patient group, or in individuals who score high on some individual differences measure. In these cases, you will probably need to equate across conditions or participants as well as across bins/classes (e.g., by checking the box labeled Across BESTsets).

This can sometimes require excluding a very large number of trials. For example, imagine that you ran a study to compare location decoding accuracy for Group A and Group B, and 48% of trials were rejected for one participant in Group A, but fewer than 5% were rejected in each of the other participants. If you simply loaded all the BESTsets and checked the box labeled Across BESTsets, eStudio would exclude 48% of the trials for every participant in both groups. This would make you very unhappy, because throwing out so many trials will reduce your statistical power. Here’s a trick to make you happier: You could instead equate the number of trials across individual pairs of participants. That is, you would randomly pair together one participant from Group A with one participant from Group B, load their BESTsets and check the box for equating Across BESTsets, and decode those two participants. You’d then repeat this for every other pair of participants. That way, there would be only one pair where 48% of the trials would need to be excluded, and yet the number of trials per bin/class would be equated on average for the two groups.

We’re now ready to perform the decoding of participants 301–305. If you have a reasonably fast computer or you are willing to wait a while for the decoding to finish, go ahead and click the Run button in the decoding panel. If you don’t want to wait, you can go to the MVPCsets panel, click Load, navigate to the MVPCsets folder, and load the MVPCsets that I’ve already created for these participants (which are named 30#_ID1vsID2.mvpc).

When you have all five MVPCsets loaded (301-305), make sure they are all selected in the MVPCsets panel. You should see them all overlaid as in Screenshot 12.5 (but they may look a little different given the randomization that is used in the process of assigning trials to averages/pseudotrials). You can see that there is a lot of variability, mainly due to the noise in the single-participant EEG data.

Now let’s make a grand average of these five MVPCsets. Make sure they are all selected in the MVPCsets panel and then open the Average Across MVPCsets panel. You should see that the MVPCsets text box contains 1:5, which indicates that MVPCsets 1-5 will be averaged together. Make sure that the option for computing the point-by-point standard error is selected, and click Run. Name the resulting MVPCset whatever you’d like. You should now see the grand average decoding waveform in the main plotting window, with the standard error indicated by the shaded regions around the lines.

---