Comparison of Visual and Auditory Modalities for Upper-Alpha EEG-Neurofeedback *

Electroencephalography (EEG) neurofeedback (NF) training has been shown to produce long-lasting effects on the improvement of cognitive function as well as the normalization of aberrant brain activity in disease. However, the impact of the sensory modality used as the NF reinforcement signal on training effectiveness has not been systematically investigated. In this work, an EEG-based NF-training system was developed targeting the individual upper-alpha (UA) band and using either a visual or an auditory reinforcement signal, so as to compare the effects of the two sensory modalities. Sixteen healthy volunteers were randomly assigned to the Visual or Auditory group, where a radius-varying sphere or a volume-varying sound, respectively, reflected the relative amplitude of UA measured at EEG electrode Cz. Each participant underwent a total of four NF sessions, of approximately 40 min each, on consecutive days. Both groups showed significant increases in UA at Cz within sessions, and also across sessions. Effects subsequent to NF training were also found beyond the target frequency UA and scalp location Cz, namely in the lower-alpha and theta bands and in posterior brain regions, respectively. Only small differences were found on the EEG between the Visual and Auditory groups, suggesting that auditory reinforcement signals may be as effective as the more commonly used visual signals. The use of auditory NF may potentiate training protocols conducted under mobile conditions, which are now possible due to the increasing availability of wireless EEG systems.


I. INTRODUCTION
Neurofeedback (NF) refers to the real-time self-regulation process, during which an individual is presented with a representation of a feature of interest of their own brain activity, so that they can consciously control it [1].During the process, individuals become aware of the variations occurring in their brain activity in real-time and are able to adapt in order to achieve optimal performance [2].The underlying premise of NF is that through operant conditioning training and neuroplasticity mechanisms, an * Research supported by Portuguese Science Foundation (FCT) through Grant LARSyS UID/EEA/50009/2013.
Athanasios Vourvopoulos is with the University of Southern California, Los Angeles CA, USA.
Floriana Acoto is with the University of Salento, Lecce, Italy Sergi Bermúdez i Badia is with the Madeira Interactive Technologies Institute and with the Faculdade de Ciências Exatas e da Engenharia, Universidade da Madeira, Funchal, Portugal individual is able to control the brain activity in the desired direction, by inducing long-term training effects [3][4].NF training is appealing because, unlike other methods that aim to influence brain activity, it is non-invasive and does not prompt dependency on outside sources [1,4].Owing to its high temporal resolution and portability, EEG is the most commonly used technique for NF [5].
Despite the success and potential of EEG-based NF training, namely in the improvement of cognitive function or normalization of aberrant brain activity in disease [5], a variety of procedural and theoretical factors remain unclear or smudged by disagreements in current literature [6].One such factor is the choice of the sensory modality utilized for the generation of the NF signal, which is often grounded on practical reasons or participants' specific characteristics [5].The chosen NF signal modality may be a key factor for a successful training [6], and its influence on different training protocols and on the outcome of NF is yet to be thoroughly assessed and documented.In fact, it is not rare that, when NF studies are designed, little care is given to how the elements of the feedback signal may impact the individuals' ability of regulating brain activity [7].Moreover, the type of NF signal is oftentimes poorly detailed, which denounces the lack of importance given to this component of NF systems [8].Typically, NF applications concentrate on mapping the EEG parameter of interest directly onto audiovisual stimulus components or, when targeted towards children, onto more complex and attractive scenarios [7].
Importantly, there has been recently an increase in the availability of portable EEG systems [9] and great efforts have been made to perform EEG recordings outside the usual constrained laboratory settings [7,[10][11].In order to conduct NF studies under mobile conditions, the auditory sensory modality might be much more appropriate than a visual display of the NF signal.
Few are the studies that have investigated the effects of different NF sensory modalities on the effectiveness of NF training.The visual sensory modality is most frequently explored in NF training protocols, even though the auditory modality has also been utilized and often in combinations of the two [7].The work by Fernández et al. (2016) [8] is, to our knowledge, the only study that has compared directly the two sensory modalities, reporting superior results for the latter.Nijboer et al. (2008) [12] and Hinterberger et al. (2004) [13] have also compared the two modalities, but in the context of brain-computer interfaces (BCI).Both reported superiority of the visual modality, although learning in the auditory group was still reasonably attained.
In this study, we compared directly the visual and auditory sensory modalities in terms of the effectiveness of an EEG-based NF-training system targeting the upper-alpha (UA) band and working memory enhancement.The implemented protocol targeted the UA band of the EEG signal, which has been shown to be independently trainable and to be associated with working memory enhancement [14][15][16][17].A central scalp location (electrode Cz) was chosen as a compromise between the predominantly posterior distribution of alpha band power and the frontal lobe involvement in working memory [18].

1) Participants and study design
A total of 16 healthy right-handed participants with no previous NF experience, and with normal or corrected to normal vision, were randomly assigned to one of two groups, of 8 participants each: the visual group (VIS) (4 males; ages (years): 22.5 ± 2.7) the auditory group (AUD) (3 males; ages (years): 22.9 ± 1.2).Participation was voluntary and no monetary compensation was offered.An informed written consent was obtained from participants after they were duly informed about the entire procedure, objectives, possible side effects and exclusion criteria.The experimental procedures involving human subjects were performed according to the Guidelines of the Declaration of Helsinki with the approval of the Institution's Ethical Review Board.
Before training, participants were asked to complete selfassessment health-related questionnaires (36-Item Short Form Survey and Hospital Anxiety and Depression Scale) for screening.Working memory was assessed before and after training, by two tasks: digit span (forward and reverse) and nback (2-back and 3-back), implemented in Presentation® software (Version 20.1, Neurobehavioral Systems, Inc., Berkeley, CA, www.neurobs.com).
Both groups were submitted to the same experimental design, the only difference between them being the NF signal sensory modality.Training consisted of 4 sessions, on 4 consecutive days, at approximately the same time of the day.In session 1, participants received all the required information concerning their participation and were asked to sign the informed written consent, fill in the health-related questionnaires and complete the first set of working memory tests.Subsequently, the NF training session began.
Participants were asked to remain as still as possible, avoiding body or head movements, and avoid frequent eye blinking during trials, availing of the pauses to do so if necessary.Eye closure or falling asleep were not allowed.In the second and third sessions only the NF training was performed.In the fourth and last session, after training, participants performed the second set of the working memory tests.
Each training session began with a 4-minute baseline (pre-baseline), alternating 1-minute periods with eyes open (EO) or eyes closed (EC), during which subjects were instructed to stay quiet and passively let their thoughts flow, while fixating their gaze on a white cross on the screen or close their eyes, respectively.Subsequently, the NF training period began.This period was divided into 5 sets composed of 3 blocks each, which, in turn, consisted of 2 1-minute trials.In between blocks there was a pause of 15 seconds and in between trials the pause was 10 seconds long.The total training time was about 40 minutes.In the end, a baseline identical to the pre-baseline was also recorded (postbaseline).

3) Online data processing
The NF protocol targeted the increase of the relative amplitude of the upper alpha (RAUA) in channel Cz, as defined in (1) [23].Considering the large inter-individual variability of the UA band, this was determined individually for each participant based on the respective pre-baseline data of the first session [19].The power spectral density (PSD) of the EEG signal recorded during both EO and EC periods was computed using the Welch method [20] in MATLAB (R2016b, MathWorks), with a window length of 5 s, an overlap of 10% of the window length and the number of discrete Fourier transform points equal to the size of the window.The frequencies at which the EC and EO spectra intersect are the lower transition frequency (LTF) and higher transition frequency (HTF), respectively, and define the lower and upper boundaries of the individual alpha band (IAB).The individual UA (IUA) band is then defined as the frequency interval between the peak frequency of the EC spectra, known as individual alpha frequency (IAF) [19,21], and HTF.
During the training periods, the learning parameter, RAUA, was computed in real-time in OpenViBE according to [22], as the sum of amplitudes in IUA divided by the total sum of amplitudes, between 4 and 30 Hz: where () is the frequency amplitude spectrum at frequency , calculated by means of a sliding window fast Fourier transform (FFT), and ∆ is the frequency resolution.A 2 s sliding window epoched the signal, shifting every 0.125 second.On each incoming window, the spectrum amplitude was obtained by applying the FFT, and two frequency bands, UA and 4-30 Hz, were selected and averaged across all contained frequencies.Finally, a moving average was applied, by computing the average over the last 10 epochs received, and the RAUA at Cz was computed.

4) NF signal
The NF signal (RAUA at Cz) was forwarded from OpenViBE to the Unity game engine (UnityTechnologies, San Francisco, California, USA), via the lab streaming layer (LSL) system.
The visual NF signal was displayed as a sphere of varying radius that changed between red and white, set over a simple horizon line background.The radius of the sphere increased linearly with the NF signal.Additionally, the sphere's color interchanged between red and white depending on whether the NF signal was below or above the predefined threshold for the session, respectively.Participants were instructed to keep the sphere as large as possible in addition to keeping it white for as long as possible.To avoid rampant fluctuations from very large to very small sizes, which could be counteracting, the sphere's radius varied more slowly, but still linearly, when the NF signal was below the threshold.
The auditory NF signal consisted of a sound of varying volume that alternated between white noise and piano music.
Participants were asked to keep their eyes open, while fixating the gaze on the screen, which displayed the same horizon background as in the visual feedback, but no sphere was displayed.Whenever the NF signal was below the predefined threshold, a white noise sound arose and its volume increased exponentially the lower the feedback values below the threshold.Otherwise, a piano music (a loop of a continuous 30-second-long segment with no silent periods) appeared with a volume that increased exponentially with the NF signal, similar to the feedback signal of [23].Both sounds were adapted to be volume matched.
Determining the fixed range of values between which the NF signal was expected to vary was important to settle the maximum minimum size/volume of the sphere/sound.These were set to correspond to the 1 st percentile and 99 th percentile plus 20% of its value, respectively, of the EO period RAUA values of the pre-baseline of session 1.The addition of 20% avoided the saturation of the sphere/sound on the maximum size/volume.The first session's pre-baseline signal was also used to define the reward threshold, i.e. the level at which positive feedback (white vs. red sphere / piano music vs. noise) is provided.In session 1, the threshold was set to be equal to the median RAUA during the EO prebaseline period.In the following sessions, the threshold was updated according to the participant's performance in the previous session.If the percentage of time spent above the threshold in the previous session exceeded 60% then the new threshold was increased by 5 percentiles with respect to the same EO period in the next session.Additionally, if there was a constant increase from set to set, of the RAUA, then the threshold was incremented by 10 percentiles.Otherwise, if that percentage was below 40%, the new threshold was decreased by 5 percentiles.

5) Offline data analysis
After data acquisition was completed, EEG data were analyzed using built-in functions of MATLAB and of its toolbox EEGLAB.The raw data were band-pass filtered between 4 and 30 Hz, and then re-reference to the average reference.Time-frequency decomposition was then performed using a Morlet wavelet transform with wavelet factor 7 [24].A moving average was then applied for smoothing signal fluctuations, as was performed in the online signal processing.The relative amplitude (RA) of each of the following frequency bands was calculated in an equivalent way to that of (1), where () now corresponds to the Morlet wavelet transform' frequency amplitude spectrum: theta (4Hz -LTF); Lower alpha (LA: LTF -IAF); UA (IAF -HTF); and beta (HTF -30 Hz).
The training progress was evaluated in terms of the variation of RAUA in Cz within and across the training sessions, assessed using the following metrics [22]: -Intra-session slope, averaged across sessions: where   is the slope of the linear regression that fits the evolution of the learning parameter along the means of all 5 sets in session .Non-learners were identified as those individuals for which IntraS was negative.
-Inter-session amplitude change: where   is the average RAUA in set ; this is the difference between the means of the last two sets of the last session (S 4 ) and the means of the first two sets of the first session (S 1 ), relative to the latter.
To investigate target band independence, as well as topographical specificity of NF training [15,[25][26], the learning indexes were calculated not only for the RAUA and channel Cz, but also for the frequency bands theta, LA and beta, as well as for all other recorded channels.
Given the small sample size and the fact that normality was not guaranteed for all the variables (assessed with Shapiro-Wilk W-test), non-parametric tests Wilcoxon Signed Rank and Wilcoxon Rank Sum were used for statistical analysis of the results.Right-tailed tests were employed when there was an expected increase from pre-to post-conditions.Correlation between variables was assessed with Pearson or Spearman correlation.Multiple comparisons correction were required when computing statistical analysis on topographical maps, for which the false discovery rate method was used [27].The significance level was set to 5% (p < 0.05), for all statistical comparisons.

III. RESULTS
Significant effects of NF training were found on the target RAUA at channel Cz, for both VIS and AUD groups, mostly within but also between sessions.One participant in the VIS group and two in the AUD group were classified as nonlearners.Only the AUD group showed a significant improvement in working memory with NF training, in the reverse digit span test (p = 0.03).No significant differences between groups were found in working memory performance.

1) Training Effect on RAUA in Cz
The evolution of RAUA at Cz across the 5 sets of each session is shown in Fig. 1, for the median of all participants and for learners only, in each group.RAUA clearly increases within each session and a small overall increase can also be observed between the first set of S1 and the last set of S4.The median of the first set of each session is overall higher than the median of the first set of the previous session, which suggests a subtle carryover effect from session to session.
In the AUD group a slight decrease from the fourth to the last sets is visible, particularly in the first 3 sessions.

2) Training Effect on Other Frequency Bands
The distributions across all learners of the indexes IntraS and InterA for all frequency bands, in each group, are shown in Fig. 2. IntraS was significantly different from 0 for the UA and theta bands in both groups, being positive for the UA and negative for the theta band.IntraS of LA was significant only in the VIS group.None of the indexes were significantly different from 0 for the beta band.With regards to InterA, it was significantly different from zero only for UA in both groups.
Significant positive Spearman correlations were found between set number and mean RAUA in S1 to S3 of the VIS group and S2 and S4 of the AUD group.Significant negative Spearman correlations were found between IntraS of the theta and LA bands, between IntraS of theta and UA bands and between InterA of theta and LA bands.

3) Training Effect on Scalp Locations
The topographical distribution of the RA of the different frequency bands in the beginning and end of the NF training period is shown in Fig. 3 for each group.In the VIS group, training lead to a decrease of the RA of the theta band, in central and more posterior regions.Concerning the LA band, an increase was observed across the whole scalp.As for the UA band, the increase was more visible in parietal and occipital regions, while beta showed a decrease in right temporal areas.In the AUD group, theta distribution was similar to that of the VIS group in both conditions.LA presented a different distribution from that of the VIS group, with more localized increases on central and frontal areas, in addition to occipital.UA's increase was also more localized in occipital areas than in VIS group.Concerning progress within sessions (not shown), in the VIS group, an increase in LA and UA was found mainly in posterior regions but spreading to central and frontal areas.The theta band decreased in central and posterior regions, while beta decreased in temporal regions.In the AUD group, the same pattern was observed, except for LA, which was more centrally/frontally located.
The topographies of the p-values assessing each learning index (IntraS and InterA) against 0 are presented in Fig. 4, for each frequency band and each group.Although uncorrected for multiple comparisons, these provide an enlightening perspective on the spatial distribution of the learning effects.In the VIS group, uncorrected significant IntraS values are seen mainly for the theta band in central regions and for the lower and upper alpha bands in central-parietal and occipital regions, while the beta band did not produce significant values.InterA presented significant uncorrected p-values in frontal-central and occipital areas particularly for the alpha bands.In the AUD group, similarly to the VIS group, IntraS for the theta band has a greater concentration of significant effects in the regions near the Cz channel, while LA also concentrates in occipital regions.As for the UA, the distribution involves more parietal-occipital areas, in addition to central leads, while beta did not produce significant values.Concerning InterA the learning effects concentrate in central and occipital areas for UA, in occipital and temporal-central regions for LA and in parietal and occipital areas for beta.No significant differences were found between groups when applying the two-sided Wilcoxon Rank Sum test, for none of the indexes in any of the bands.

IV. DISCUSSION
A NF-training system was developed and implemented to compare the effectiveness of visual and auditory modalities of the NF signal on a protocol targeting the UA band and working memory enhancement.The protocol was tested on a group of healthy participants, randomly assigned to the VIS or AUD groups, where a sphere varied in size (and color) and sound varied in volume (and content), respectively.Both groups were able to up-regulate their UA at Cz, more significantly so within than across sessions.

1) Training Effects on UA in Target Location
A learning effect was observed for both VIS and AUD groups, as they were capable of voluntarily increasing their RAUA over training, as reflected in an overall increase of the relative amplitude of this band within and across sessions.There was a more noticeable learning effect within than across sessions, particularly in the VIS group.Our findings are consistent with those of Hanslmayr et al. (2005) [14] and Escolano et al. (2012) [28], who also reported significant enhancement of UA within the single session.In contrast, Cho et al. (2008) [29] and Escolano et al. (2014) [30] found effects on alpha and UA-training across but not within sessions.Moreover, Zoefel et al. (2011) [15] also found longterm increase of the UA throughout sessions and reported evidences of each training session building upon the previous ones.
These discrepancies may be partly explained by the differences across studies regarding the number and duration of the training sessions.In fact, some studies employ less intensive designs, with a larger number of shorter sessions, compared to ours.For example, Dekker et al. (2014) [31] used 15 sessions with a training time per session that was about 1.5 times lower than ours (~24 minutes) and found a linear increase in UA over the first 10 sessions, after which a decreasing trend emerged.These results suggest that the subtle effects found across-sessions in our study may be due to the relatively small number of training sessions (four).Importantly, Dekker et al. (2014) found a decreasing pattern from second to third (and last) trial, common to all 15 sessions, which was also hinted by the within-session results in our AUD group, between the fourth and fifth sets.These results may be due to exhaustion or reduced motivation as the upcoming weariness is foreseen.In order to overcome this, session duration could be reduced in future studies.

2) Training Effects on Other Frequency Bands
We found a decrease in the theta band that was correlated with the increase in the UA band.For example, the IntraS learning index for the theta band was only positive for those individuals who were considered UA non-learners: those who were not capable of increasing UA within sessions were also not capable of decreasing the theta band.This dichotomy between the UA and theta bands is in agreement with the findings of Klimesch in 1999 [19].Hanslmayr et al. (2005) [14] suggested possible interactions between these two bands, such that one is capable of inducing changes in the other.However, their results showed that only about half of UA responders were also considered theta responders.
In our study, the LA had a similar increasing behavior as the UA, although not all UA non-learners would be considered LA non-learners.In another study, Nan et al. (2012) [16] also reported changes in bands other than the trained IAB, namely in the UA and LA, but also in delta and sigma bands.However, UA and LA bands are thought to reflect different cognitive processes [32].In fact, Dekker et al. (2014) [31], for example, reported different NF outcomes in the two bands: power decreased by the end of each individual session, and also by the end of the sequence of sessions, only for UA and not for LA.
More generally, Zoefel et al. (2011) [15] demonstrated that the UA could be trained independently of other frequency bands, since the immediate neighboring bands were not significantly affected during their training protocol.In contrast, Escolano et al. (2014) [30] reported within sessions increase of upper beta and decrease of LA and theta, in EO task-related activity and training, while the effects along sessions were positive for the trained parameter (relative UA power in fronto-central sites) in task-related activity and negative for delta power.According to the criteria defined by Zoefel et al. (2011) [15], one cannot affirm total independence of UA training regarding our study.However, it is remarkable that the flanking beta band was not affected and that the LA band was only affected in the VIS group but not in the AUD group.These results suggest that auditory feedback might promote training independence more than visual feedback.

3) Training Effects in Other Scalp Locations
Our results demonstrated that training had greater effects in the UA band, particularly in central locations and to some extent also in more posterior regions.This is consistent with the fact that alpha activity is usually more predominant in these regions.Our results are also consistent with the study by Van Boxtel et al. (2012) [33], which demonstrated that NF training of alpha on central sites was associated with increased posterior alpha activity.Another study, by Escolano et al. (2014) [30], also observed across sessions effects on UA (and also theta), not only in the trained location (fronto-central sites) but also in parieto-occipital sites, during task-related activity.Within sessions, the trained parameter decreased, but a relative power increase was seen for upper beta in parieto-occipital sites and also an absolute power decrease in theta and LA, during task-related activity.In Egner et al. ( 2004)'s investigation [26], training of theta/alpha ratio in "posterior scalp regions was associated with decreased frontal beta band activity".Hanslmayr et al. (2005) [14] found differences in UA power before and after UA training in the right parieto-occipital areas for responders.
In Fernández et al. (2016)'s work, the feedback location was not fixed, as it corresponded to the lead presenting the highest abnormal ratio of the target frequency bands, theta/alpha value.While the auditory group revealed increased frontal alpha, in the visual group this increase was observed in frontal as well as parieto-occipital regions.Additionally, centro-parietal beta increased only for the auditory group.Such extreme topographic differences between groups were not observed in our work, except for LA, which was apparently more centrally located in the AUD group compared to the more posterior location in the VIS group.

4) Visual vs. Auditory Feedback Signal
We found only subtle differences between the NF training effects in the two groups, with the auditory feedback apparently yielding more spatially specific UA effects compared with the visual feedback.Few are the studies that directly compare visual and auditory feedback modalities.A search of the relevant literature revealed only one study, Fernández et al. (2016) [8], where the modalities were compared in NF training performance.In that study, the auditory modality proved to be superior to the visual modality.The authors identified the higher contingency of the auditory stimulus, which reaches the brain faster than the visual, as a possible explanation as to why learning in the auditory group was better.In related studies on BCIs, both Hinterberger et al. ( 2004) [13] and Nijboer et al. (2008) [12] found superiority of the visual modality with regards to BCIperformance, even though learning was still attained with auditory feedback.Competition for attentional resources and possibly distracting "harmonies and melodies" were identified as potential reasons for the reduced auditory performance.

5) Limitations and future work
The major limitation of our study is the relatively small size of the study sample which may have rendered insufficient statistical power to detect differences between groups.Further research is thus required to determine if and under which conditions one of the sensory modalities is effectively better to enhance NF training performance.
The lack of statistical power probably also explains the insignificant improvements in working memory observed for most tests between pre-and post-training conditions, except for the AUD group, in the reverse Digit Span.In Nan et al.
(2012)'s study [16], short-term memory increases in forward and reverse tests of the alpha NF group were significantly higher when compared to the control group.The authors discarded the possibility of test practice effect, considering the control group also reported improvements, although not significant.Significant improvements in cognitive performance following UA training have also been observed in previous studies [14,15,30,[34][35][36].
One aspect that may explain the poor working memory performance improvement in our study is that we may have underestimated the number of non-learners.In fact, about 12.5% of the participants in the VIS group and 25% in the AUD group were classified as non-learners.These percentages fall somewhat below those found in literature for similar protocols and might be indicative that non-learners have been overlooked.The choice regarding the criterion for selecting learners could be further sustained in the future, for example, by evidence of learning (positive slope across session) in at least 3 of the 4 sessions.
Another aspect that could be improved in our NF systems is that the elements of the feedback display remained unchanged for all participants.To improve trainability, in the future these elements could be further tailored to each individual participant, for example, by allowing them to choose the preferred colors of the sphere or the sounds.Additionally, considering the known differences in the alpha power between eyes-open and eyes-closed EEG, the existence of a third training group receiving auditory feedback with eyes-closed (similar to the auditory group in [8]) would be interesting to assess the potential trainability differences of training under one condition or the other.

Figure 1 .
Figure 1.Median RAUA at Cz across all subjects (top) and learners (bottom) for each group, along the 5 sets of each session (S1 to S4).

Figure 2 .
Figure 2. Distribution across learners of the learning indexes at Cz for all the relative amplitude of the four frequency bands at study.* mark indexes significantly different from 0 (p<0.05)resulting from the Wilcoxon Signed Rank test (right-sided for UA; two sided for the other bands).

Figure 3 .
Figure 3. Topographical display of the median relative amplitudes in the beginning (Pre) and end (Post) of all training sessions, across the learners of the VIS (top) and AUD (bottom) groups, for each frequency band.Pre: mean of the first two sets of session 1; Post: mean of the last two sets of sessions 4.

Figure 4 .
Figure 4. Topograpical distribution of the p-values resulting from the two-sided Wilcoxon Signed Rank test (right-sided for UA in location Cz) of the indexes IntraS and InterA, for the learners of the VIS (top) and AUD (bottom) groups, uncorrected for multiple comparisons.