12.9: Exercise- Temporal Generalization

You can learn a lot about how specific neural representations come and go over time by looking at what’s called temporal generalization of decoding (King & Dehaene, 2014). The basic idea is that you train a decoder at one time point and then you feed it test cases from another time point. If the decoding accuracy is above chance, this suggests that similar brain representations are used at the different time points. For example, if you train the decoder at 150 ms and test it at 600 ms, and you find significant decoding, this suggests that the sensory representation at 150 ms is still active at 600 ms (or, to be more precise, that the representation at 600 ms is similar to the representation at 150 ms).

You can get information about temporal generalization in eStudio by simply checking the box labeled Temporal general matrix when setting up the decoding in the Multivariate Pattern Classification panel. In fact, you already did this in the previous exercise. When a decoder is trained with the data from a given time point, this option will cause it to test that decoder with the data from every time point. This gives you a decoding accuracy value for every combination of training time point and testing time point.

Let’s see what this looks like. Make sure you still have the 9 MVPCsets from the previous exercise loaded. Then select the MPVCset labeled 3_N170, go to the Plot/Export Temporal Generalization Matrix panel, and click the Plot button. You should see something like Screenshot 12.14A. The vertical (Y) axis indicates the time point used to train the decoder, and the horizontal (X) axis indicates the time point used to test the decoder. Every combination of training time and testing time is shown. The color scale shows the decoding accuracy, with lighter values indicating higher decoding accuracy.

From approximately 100-400 ms, decoding accuracy is well above chance (marked in bright yellow) mainly along the diagonal. This means that training at a given time point (e.g., 200 ms along the Y axis) produces good decoding only for that point and the surrounding time points (e.g., 190-210 ms along the X axis). In other words, there was very little temporal generalization in this time range, which means that the representational format changed quickly from moment to moment. After 400 ms, however, there was more generalization. For example, when the decoder was trained at 460 ms (along the Y axis), there was good decoding (yellow values) when the decoder was tested at most points between 450 and 700 ms (along the X axis). This suggests that the later representations tended to persist for hundreds of milliseconds. The pattern shown by this participant is fairly common.

You can see another common pattern by selecting the data for 7_N170 from the MVPCsets panel and plotting the temporal generalization matrix for this subject (see Screenshot 12.14B). This subject also shows a line of bright yellow along the diagonal from approximately 100-400 ms (but with a little more generalization from 300-400 ms than was seen for 3_N170). However, this was accompanied by horizontal and vertical lines of bright yellow at approximately 100 ms. That is, when the decoder was trained at 100-120 ms (along the Y axis), good decoding (yellow values) could be seen from approximately 250 to 400 ms (along the X axis). Conversely, when the decoder was tested at 100-120 ms (along the X axis), good decoding could be seen for decoders trained at many time points from approximately 250-700 ms (along the Y axis). In other words, the representations present from approximately 100-120 ms were similar to those present at much later time points for this subject.

Although I am showing you these two patterns in two subjects from the same experiment, these two patterns are usually observed in grand averages from different experiments. You can look at the temporal generalization matrix from this experiment by selecting the grand average MVPCset and plotting it using the Plot/Export Temporal Generalization Matrix panel. To perform statistical analyses, you can export the single-subject temporal generalization matrices using the Export button in this panel. The Export tool allows you to save the plots of the matrices in files on your computer drive, the matrices themselves in spreadsheets on your computer drive, or both.

---