4.12: Background- Quantifying Signal, Noise, and Waveform Distortion

Defining and Quantifying the Signal

We define the signal in terms of the amplitude or latency score you are trying to obtain in a given study. In the context of Figure 4.8 from the previous section, this might be the mean amplitude from 300-500 ms for the P3b component (Figure 4.8BD), or it might be the latency of the most negative voltage between 150 and 200 ms for the N170 component (Figure 4.8AC). You might be surprised that we are defining the signal in terms of relatively abstract scores like these rather than something like the amplitude of the raw EEG. However, these scores are what you will be putting into your statistical analyses and using to test your scientific hypotheses, so these are the signals you care about. And, as illustrated in Figure 4.8, a given filter can have very different effects on different ERP components, so we want to define the signal in a way that reflects the component we will be scoring.

Figure 4.9 illustrates another important reason to use the score as the signal. The top panel in this figure shows the N400 difference wave (unrelated minus related) from the ERP Core N400 experiment that was the focus of Chapters 2 and 3. The bottom panel shows the same waveform with high-frequency noise added (the kind of noise you would get from muscle activity). 

Imagine that you wanted to quantify N400 amplitude as the most negative voltage between 300 and 500 ms (the voltage of the N400 peak). The high-frequency noise could have a big impact on the score you obtain. Now imagine that you decided to quantify N400 amplitude as the mean voltage between 300 and 500 ms. With this scoring method, the rapid upward and downward voltage deflections (“uppies” and the “downies”) in the high-frequency noise will largely cancel out, so the high-frequency noise will have only a minimal impact on the N400 score. Consequently, applying a low-pass filter to the data prior to scoring will improve the N400 peak amplitude score but will have little impact on the N400 mean amplitude score. This shows yet again that the optimal filter settings will depend on the specific score we are obtaining in a given experiment.

Defining and Quantifying the Noise

Now that we’ve defined the signal, we’re ready to define the noise. You might be expecting the noise to be defined in terms of the raw EEG. However, you just saw in Figure 4.9 that the effect of a given type of noise depends on how you are scoring the ERP waveform. To take this into account, we’re going to define noise in terms of any factors that cause our scores to be imprecise. 

Precision is classically defined in terms of whether you get the same value every time you try to measure something. For ERPs, a score is precise to the extent that we would get the same score every time we repeated the experiment in a given participant (assuming no learning or fatigue). If we repeat the N400 experiment 100 times in a given participant, measure the N400 amplitude for each experiment, and find that our N400 amplitude score is between 7.1 and 7.3 µV for all 100 repetitions, that would be a very precise score!

This is what the standardized measurement error (SME) is designed to quantify. You may recall that the SME was introduced in Chapter 2.10 (see Luck et al., 2022 for a detailed description). Whenever you compute averaged ERPs in ERPLAB, you will get the SME for a default set of time windows (e.g., 300-400 ms). This SME value reflects the imprecision of the mean voltage score from a given time window. You can also specify a custom time window that matches the measurement window you’re planning to use in a given experiment (e.g., 300-500 ms for the N400 experiment in Figure 4.9).

When you are scoring the amplitude of an ERP component as the mean voltage in a specific time window, it’s trivial for ERPLAB to compute the SME using an analytic approach. We call this the analytic SME or aSME. It’s also possible to compute the SME for other scores, such as peak amplitude or peak latency. However, this requires a much more complicated approach called bootstrapping, which leads to the bootstrapped SME or bSME.

No matter whether you are using aSME or bSME, the SME value is a great way of quantifying the noise in your data in a way that reflects the extent to which the noise impacts the specific amplitude or latency scores in your study. For this reason, we use the SME as our metric of noise when defining the optimal filter.

Bottom line: The signal is defined as the score for a given study, the noise is defined as the SME of that score, and the signal-to-noise ratio is the score divided by the SME of that score.

Defining and Quantifying Waveform Distortion

Now let’s consider what we mean by “unacceptable levels of waveform distortion” and how to quantify it. I have to admit that this part is more subjective, because what is acceptable in one context might be unacceptable in another. At a minimum, the threshold for “unacceptable” is likely to differ across research areas and participant populations. But we did our best at coming up with something that could be easily quantified and is likely to work well in many situations.

Filters produce two main types of waveform distortions. First, low-pass filters may cause the filtered waveforms to begin earlier and end later than unfiltered waveforms (see Screenshot 4.9 in section 4.4 or Screenshot 4.22 in section 4.9). In most areas of research, this is not a major problem, because researchers are typically interested in the difference in timing across groups or conditions, not the absolute onset or offset time of an effect. For example, if a filter shifts the onset time of the ERPs 20 milliseconds earlier in both Group A and Group B, this won’t typically change the difference in latency between Group A and Group B. Occasionally, it is useful to examine the absolute latency at which a difference wave first deviates from zero. In these cases, you can simply quantify the amount of latency shift produced by the filter and take that into account (see, e.g., Bae & Luck, 2018). But in most cases, the change in ERP onset and offset times will not lead to incorrect conclusions.

The main filter artifacts that are likely to lead to incorrect conclusions are the artifactual peaks that can be produced by high-pass filters. You saw an example in the opening of this chapter, and you saw additional examples in section 4.10. Figure 4.10 below provides another example, showing the effects of applying a high-pass filter with different cutoffs to a simulated N400 effect. With a cutoff of 0.1 Hz, the filtered and unfiltered waveforms are nearly identical. If we increase the cutoff to 0.5 Hz, we can see a distinct positive peak preceding the negative N400 wave in the filtered waveform. This artifactual positive peak is even bigger with a 1 Hz cutoff, and with a 2 Hz cutoff the artifactual positive peak is almost as large as the negative peak of the actual N400. If you used a filter of 0.5 Hz or higher in a real N400 experiment, you might think the earlier positivity is a real component, leading to completely incorrect conclusions. I think almost everyone would agree that a distortion that leads to an incorrect conclusion is an “unacceptable level of waveform distortion.”

In our procedure for determining optimal filter settings, we came up with a simple way of quantifying this kind of artifactual peak. We simply measure the size of the artifactual peak and express it as a percentage of the size of the real peak. We call this the artifactual peak percentage or APP. The APP is shown for each of the filter settings in Figure 4.10. It is only 0.5% for the 0.1 Hz filter, but it jumps to 10.6% for the 0.5 Hz filter. 

For reasons that are described in Zhang et al. (2024a), we obtain the APP from simulated ERP waveforms like those in Figure 4.10. In the next exercise, you will see how you can create these simulated waveforms in ERPLAB and have them approximate the true waveforms in your data. After filtering a simulated waveform, you can use the Measurement Tool panel to score the peak amplitude of the true peak and the artifactual peak in the filtered waveform. You’ll then take the absolute value of these two peak amplitudes (to make sure we end up with a positive APP value) and then compute: 

APP = 100 x absolute value of the artifactual peak amplitude ÷ absolute value of the true peak amplitude

Once you have quantified the waveform distortion as the APP, you will need to decide whether the APP is acceptable or unacceptable. This is where things get subjective and context-dependent. How much distortion is too much?

For the kind of research my lab typically conducts, where we obtain very clean data from highly cooperative participants, we decided that an APP threshold of 5% would be reasonable. That is, we avoid any filters with an APP of >5%. This would be like a 0.5 μV artifactual negative peak preceding a 10 μV P3b component or a 0.1 μV artifactual positivity preceding a 2 μV N2pc. These artifactual peaks are small enough that they would just look like noise and would not be statistically significant in the kinds of experiments we run. By contrast, if we used a filter that produced an APP of 10%, this would be like a 1 µV negativity preceding a 10 μV P3b component or a 0.2 μV artifactual positivity preceding a 2 μV N2pc. These effects might be statistically significant and therefore lead to incorrect conclusions. For these reasons, a 5% APP threshold seems reasonable in my lab’s research (both our basic science research using college student participants and our translational research on schizophrenia using a broad range of patients and control subjects).

However, a 10% threshold might make more sense in other areas of research. For example, studies of infants and young children often involve noisier data and fewer trials than the research in my lab, and a 1 µV artifactual negativity would be unlikely to reach statistical significance in a P3b study in infants. However, such studies might really benefit from using a 0.5 Hz cutoff instead of a 0.1 Hz cutoff, because the higher cutoff will reduce more of the low-frequency noise.

I can’t tell you what APP threshold you should use in your research. The higher the threshold, the more aggressive you can be with your filters, leading to less noise and more statistical power. But higher thresholds might lead to incorrect conclusions if the artifactual peaks are large enough that they might often be statistically significant. I’m hopeful that the different subareas of ERP research will, over time, reach a consensus about the most appropriate APP thresholds in those subareas.

Putting It All Together to Find the Optimal Filter

Now that you’ve learned how we quantify data quality and waveform distortion, it’s time to put everything together and describe the process of finding an optimal filter. The process is described in more detail by Zhang et al. (2024a), and you can find details of how we applied it to the 7 components of the ERP CORE in Zhang et al. (2024b). In addition, you can find example Matlab scripts at https://osf.io/98kqp/. 

Here are the basic steps of the process:

1. Find a dataset that is typical for your research in terms of the ERP component(s) you will be examining, the participant population, and the type of EEG recording system. If you don’t already have an appropriate dataset, you can probably find one online. There is a list of online EEG/ERP datasets on github, and many recent papers provide access to their data. Note that it is not usually appropriate to apply our procedure for determining optimal filter settings to a dataset and then use those filter settings to analyze the same dataset. That would be “double dipping”.
2. Create one or more artificial waveforms that simulate what the effect of interest would look like in the absence of noise. 
3. Apply many different filters to these waveforms and measure the artificial peak percentage (APP) to see which filters produce acceptable versus unacceptable levels of waveform distortion.
4. Apply many different filters to the real data and quantify the size of the signal and the magnitude of the noise after the application of each filter.
5. Select the filter settings that produce the best signal-to-noise ratio (or lowest noise level for latency scores) without producing unacceptable levels of waveform distortion.

Step 1 is up to you, but the following exercises will show you how to implement Steps 2-5 using ERPLAB Studio.