12.3: The Essence of Decoding

To understand the essence of decoding, start by imagining that Bae (2021) had only two face identities (ID 1 vs. ID 2), and we are trying to decode which one of them was presented on each trial. The decoder is given the set of voltage values across electrode sites for a trial (or the average of a set of trials) and attempts to guess whether that set of voltage values was produced by ID 1 or ID 2. A more formal term for decoding is pattern classification: a classifier (decoder) attempts to determine which class was present (ID 1 or ID 2) given a pattern of activity (the set of voltage values across electrode sites). The classifier is trained with several examples of patterns for each class, and it learns an optimal way to classify test cases that were not used for training. In a two-class problem such as distinguishing between ID 1 and ID 2, chance accuracy is 50% (i.e., this is the accuracy that would be expected by chance if the voltage patterns are the same for ID 1 and ID 2).

Decoding is conducted separately for each participant. It is also typically performed independently at each time point (e.g., 300 ms), giving us a waveform with decoding accuracy at each time point for each participant. These decoding waveforms are analogous to conventional ERP waveforms, and the statistical procedures used to analyze conventional ERPs can be applied in a straightforward manner to the decoding data.

To visualize how decoding works, we are going to start with a simplified situation in which we have only two electrodes (E1 and E2) and we are decoding from the single-trial data. As shown in Figure 12.2, these two electrode sites form a two-dimensional space, and the voltage at E1 and E2 for a single trial is represented as a location in this space. We have one set of points for ID 1 (drawn as blue dots) and another set for ID 2 (drawn as red dots). There is some overlap between these two classes, but the dots for ID 1 are mainly clustered in the upper right part of the space, whereas the dots for ID 2 are mainly clustered in the lower left part of the space. The goal of decoding is to find a decision line that optimally separates the two classes.

The classifier is first given a large number of training cases from each class, and it uses these cases to find a good decision line. However, it would not be fair to test the classifier on the same trials used for training, because the classifier might be picking up on random noise that produces small differences between the training cases for the two classes. Two test trials are therefore held out from training (marked by boxes surrounding the dots in Figure 12.2). 

Once the classifier has finished using the training data to determine a decision line, the data from the test trials are fed into the classifier, and these trials are classified according to which side of the decision line they fall on. A trial is correctly classified if it falls on the correct side of the decision line for that class. Decoding accuracy is quantified as the proportion of test trials that were correctly classified. This use of separate trials for training and testing is called cross-validation.

With just two test cases, decoding accuracy could be 0% correct, 50% correct, or 100% correct. To get more fine-grained percent correct values, we repeat the process N times (where N is the number of stimuli per class), leaving out a different pair of test trials on each repetition. For each repetition, a new decoder is trained and tested. We can then quantify the decoder as the proportion of trials that were correctly classified across all the repetitions. Each repetition is called a fold, and the whole process is called N-fold cross-validation.

Single-trial EEG epochs are usually too noisy to yield good decoding accuracy. As a result, we often create pseudotrials by averaging together multiple single trials. For example, Figure 12.2A shows single-trial data from 40 trials per class, and we could randomly divide the 40 trials for a given class into 10 sets of 4 trials, and then create an average from each set of 4 trials. This would give us 10 pseudotrials for each class, as shown in Figure 12.2B. We would train the classifier on 9 of the 10 pseudotrials for each class and then test the classifier with the 1 remaining pseudotrial for each class. We would then repeat this process a total of 10 times, each time leaving out a different pair of pseudotrials for testing. This would be a 10-fold cross-validation.

To minimize the influence of randomness, we iterate this process multiple times. On each iteration, we re-randomize which stimuli are assigned to a given pseudotrial. For each iteration, we get a proportion correct from the N-fold cross-validation, and the final decoding accuracy is the average of these proportion correct values across all the iterations. We typically run at least 10 iterations, and sometimes as many as 100.

So far, we have been assuming that you have only 2 electrode sites, giving us a simple 2-dimensional space. Fortunately, the procedure scales up in a straightforward way to larger numbers of electrodes. If you have, for example, 30 electrode sites, you will have a 30-dimensional space. Instead of finding a simple decision line in a 2-dimensional space, the classifier will find a hyperplane that optimally divides the actual 30-dimensional space, and test cases will be classified according to which side of the hyperplane they fall on. The more general term decision boundary can be used whether it is a line in a 2-D space or a hyperplane in a large-dimensional space.

We have also been assuming that you have only 2 classes, but you can also perform multiclass decoding when you have 3 or more classes. Multiclass decoding is achieved by training several individual decoders. For example, you can train one decoder for each possible pair of classes (called one-versus-one decoding). Alternatively, you can train one decoder for each class, distinguishing between that class and all of the other classes (called one-versus-all decoding). My lab has generally found that one-versus-all decoding works best with ERP data. After you’ve trained the several decoders, you can combine them with the error-correcting output codes approach (Dietterich & Bakiri, 1995). In this approach, each test case is fed into all of the individual decoders, each individual decoder indicates its confidence that the test case comes from a given class, and a single best guess is determined from the outputs of all the individual decoders.

There are many different algorithms that can be used to find the decision hyperplane. For the kinds of decoding that are common in cognitive and affective neuroscience, the support vector machine (SVM) algorithm seems to work best in most cases. It works particularly well when there are not a lot of training cases. This means that the SVM algorithm allows us to average many trials together into each pseudotrial, which creates a small number of relatively clean pseudotrials. ERPLAB implements the SVM algorithm and also allows you to use an older classification technique called linear discriminant analysis (LDA) and a newer method called the cross-validated Mahalanobis distance (Walther et al., 2016).

When applying decoding to ERPs, you will need to make several decisions: 

• You will need to choose a classification algorithm. In most cases, I recommend using an SVM with one-versus-all decoding.
• You will need to choose which electrode sites to use. Researchers typically use all sites because the classifier will naturally give the greatest weight to the most informative sites.
• You will need to choose the number of folds and the number of trials to average together for each pseudotrial. In most cases, the number of trials per pseudotrial is simply the total number of trials per class divided by the number of folds.
• If the number of trials differs from class to class (e.g., because one class is more frequent than the other), you will need to choose a method for dealing with this variability. 
• You will need to choose the number of iterations. 

As usual, it is important to make these selections a priori so that you don’t bias the results. We will discuss principles for making these choices as we go through the exercises in this chapter.

---