Beyond Research Horizons: The Synergy of Technologies in Multimodal Labs

1. The challenge of studying human behaviour 

The study of human behaviour is not a new endeavour; it has been a subject of exploration for a long time. The observation of the actions, reactions and interactions of humans is a focal point of study in many different research scenarios.

These interactions, whether occurring between individuals or prompted by external stimuli, provide information from participants' insights and environmental responses.

This analytical approach contributes to a deeper comprehension of relevant dynamics of human conduct. Some relevant fields in which these studies are performed are human factors (UX), social psychology, human-machine interactions, neuromarketing, and others.

 The challenge of studying human behaviour is well-known to researchers, given the complexity of human nature. Due to this, multimodal recording scenarios are becoming more relevant and demanded. As technology evolves, an increasing number of researchers consider adding different measurement tools in their research to provide a comprehensive source of information. Therefore, they can study human responses from several angles and analyze their interactions. Pertinent factors, such as focal points of attention or emotional states, play a significant role in studies, and these aspects can be captured as essential biological signals.  

These scenarios involve the simultaneous application of multiple sensors and measuring techniques. With them, a single test procedure gives the researcher as much insight as possible into the analysis of the subject’s reaction. However, the scope of human studies is influenced by factors like the use case, cost considerations, and resource availability.  

This article, we will explore the value these combinations of technologies can bring to the research landscape, as well as how to evaluate the best options for each scenario.  

2. How to define a multimodal laboratory?

To perform studies on human behaviour, a starting point for the researcher is to select which parameters, markers, or metrics they want to monitor on the subjects [1]. The next major factor is selecting the sensors and equipment from which they can be obtained.

The researcher's approach will define the choice of technology, since different sensors with distinct features can be used to obtain similar parameters, depending on the study case and the environment. When defining a multimodal laboratory, the key factors are the information sought, the use case, the existing constraints, and the selection of the most appropriate combination of technologies for the research.

Within this publication, we will present the signals that are especially relevant for human behaviour studies, as well as some key guidelines on how to choose between different technologies that allow for their recording. 

               a. ATTENTION FOCUS

Visual attention focus is especially relevant when evaluating human reactions and interactions with visual stimuli. It is one of the most important parameters to study in many behavioural types of research. 
 
For obtaining this information, eye-tracking systems use different technologies to measure the position and movement of the eyes (see Figure 2). More information about these technologies and the explanation behind them can be found in Learn here what are the different eye-tracking techniques and methods to record eye movements accurately, how to calibrate, and what are the limitations. | Bitbrain [2]. 
 
From those options, camera-based eye tracking allows for a non-invasive and accurate method to obtain pupil and cornea positions. They can be adapted to many different scenarios based on their specifications and designs so they can be used in multiple research environments (The Different Kinds of Eye Tracking Devices | Bitbrain[3]). That is why choosing between them depends on the preferences of the researcher (How to choose the right eye tracker for your research - Tobii[4]).

Figure 1: Eye-tracking heat map made with a screen-based Tobii eye tracker.

             b. ELECTRICAL BRAIN RESPONSE

The electroencephalographic signals (EEG) represent the electrical activity of the brain, captured by sensors. EEG is a complex signal that can be used to estimate many different conscious and unconscious responses from the user (What is EEG and what is it used for? | Bitbrain[5]). 
 
Due to the extended amount of information that can be extracted from EEG, this data is valuable in many different research scenarios, including human behaviour research labs (The EEG test: Uses, Procedures and Risks | Bitbrain[6]). As such, many types of different EEG technologies have been developed to adjust their specifications and usability to all these environments [7]. 
 
One of the main distinctions between EEG systems is how the sensors come in contact with the user’s skin:   

• Sensors that require a conductive substance as an intermediate layer before the skin. They offer the researcher a signal less susceptible to external noise, thanks to their lower contact impedance. To do so, they compromise their set-up speed and lessen the number of scenarios in which they can be applied: The Wet-EEG Cap: Semi-Dry, Saline & Gel EEG caps | Bitbrain[8]. 
• Sensors that do not require any extra substance for contact (dry sensors). This technology allows for a faster and more natural set-up, which is relevant in many human behaviour research scenarios. In exchange, they usually are more susceptible to noise than their wet-sensor counterparts. Fixed sensor layouts often require the researchers to select the EEG system based on their cortical areas of interest: How to select a dry-EEG headset for your research applications | Bitbrain[9].

Figure 2: a) Bitbrain Dry EEG (Diadem 12 ch). b) Bitbrain Semi-Dry EEG (Versatile EEG 32 ch). 

             c. OTHER PHYSIOLOGICAL INFORMATION.

Human interactions and reactions also trigger changes in other physiological parameters of the subject. Some of the most relevant for human behaviour monitoring are: 

• Electrocardiogram (ECG): measures the cardiac electrical activity of the user, including heart rate and heart rate variability.
• Electrocardiogram (EMG): measures the muscle electrical activity in any part of the body. Can be used to assess localized voluntary or involuntary movement. 
• Galvanic Skin Response (GSR): measures the changes in the skin conductivity of the user, directly related to the unconscious impact of external stimulus (by pore dilation, sweat excretion, etc.). 
• Blood Volume Pressure (BVP): records the changes in the volume of blood passing before the sensor. It can also be used to calculate heart rate and heart rate variability. 
• Respiratory effort: expresses the thorax movement and expansion due to the breathing procedure. It is used to measure respiratory rate and changes in it. 

Figure 3: Diagram of all the biosignals that can be measured through multimodal research. 

Apart from these and other types of biological parameters, conscious human reaction data (subject’s interaction with buttons or pedals, movement detection sensors, etc.) and environmental data (light sensors, temperature sensors, etc.) can also be used to further study the effects of the environment and stimulus on the user’s behaviour.

Once researchers are aware of all these possibilities, the use case of the project and the limitations that this may cause, they will be able to select which combination of biosignals can provide them with the most valuable information. 

3. Why is it useful to synchronize and study these behavioural responses?

With the implementation of multimodal recording, each individual parameter recorded gives information to the researcher by itself but also adds value to the rest of the parameters creating a synergy. For example, by tracking eye movement you can study the gaze of the participant on a display. However, if you add brain electrical activity (EEG) and skin conductivity (GSR) signals to the data collected, you can see which of the gazed-upon areas were more impactful or pleasant for the subject. 

In the same way, you can use EEG individually to study if a text is memorable, but eye-tracking (E-T) will give you the information on which specific passage of the page is the main responsible for it.

To get the most value out of the co-registration of all these signals, it is of vital importance to synchronize all these data streams. There are many different scenarios in which the time alignment of these streams is necessary for the proper analysis of the subject’s parameters. This can be relevant when finding unexpected or relevant punctual responses, as studying the synchronized data from the rest of the sensors can give the researcher information on its possible causes. For example, linking an unwanted movement to an abnormal response of an ECG signal. Also, in other cases, aggregating participants’ responses to a specific stimulus can only be done if all the signals are properly aligned with the time of the event (see Figure 1). As such, we will be able to compare how different subjects react in that specific instant, or at any given point after it. 

Figure 4: Relevance of time-accuracy on EEG event-related potential (ERP) responses between subjects. Incorrect synchronization will result in errors in post-processing stages, such as average response calculations. 

To further understand the relevance of this procedure, you can visit our previous article, EEG Sync With Other Biosensors and Software | Bitbrain[10], to get a deeper insight into its applications, especially in the field of EEG.

As explained in the article, data synchronization can be performed via hardware and software. The availability of both options gives researchers flexibility on how they want to address it, depending on their expertise on the matter or the conditions of the tests.

In that regard, in the field of multimodal recording, there are software options that automatize this procedure, like Bitbrain’s SennsLab software which allows the researcher to synchronize several devices. Other software such as Tobii Pro Lab is a tool that is compatible with other technologies to achieve a multimodal solution (Multimodal research solutions - EEG and Biometrics - Tobii[11]). 

4. Relevant research scenarios 

With the applicable parameters of the study selected, the limitations lie in the environmental conditions in which they need to be captured.

Previously, many of these signals had to be captured in unrealistic and unfavourable test conditions. Many mechanical and functional limitations determined the study design: the need to be plugged into the electrical supply, ambient light effects on the recording, wired connection between components, movement limitation for the subject, etc. These conditions were even more unavoidable in multimodal scenarios, as the use of multiple sensors usually meant the application of more constraints.

However, later technology developments have focused on the usability and accessibility of these systems. As such, there is a wider range of experimental procedures that can be selected, adding value to the studies, and increasing the relevance of the results and conclusions extracted from them.

These new research systems need to compromise between their adaptability and some of their technical specifications. That is why it is important for the researchers to select a solution that can capture the level of signal quality that they require while giving the subjects as much freedom and comfort as needed to avoid bias in their behaviour.

In this regard, we will present here two of the main human-behavior research scenarios, and some key elements to consider for finding this balance: 

a. In-lab static scenario: 

In this case, environmental conditions are usually less demanding. This allows the researchers to investigate systems with higher technical specifications for data capture: sample frequency, capture bandwidth, signal resolution, etc.

This scenario is also commonly linked to early exploratory phases, in which the spatial resolution of some of these sensors and the flexibility on their colocation needs to be higher. That way, different layouts can be evaluated to optimize the application and the experimental procedure for later more specialized phases (explained in more detail in The Wet-EEG Cap: Semi-Dry, Saline & Gel EEG caps | Bitbrain[8]). 

Some key factors to look for in these types of experiments are: 

• High spatial resolution in all sensors, as explained in the previous paragraphs. Particularly relevant for exploratory labour and studies in which a highly detailed signal is needed. 
• Flexible channel layout. Once again, it is relevant to evaluate different sensors’ dispositions and to be easily applicable in other future experiments. 
• Confortable disposition. Particularly relevant for human behaviour studies, even in-lab conditions. Reduces unwanted external conditioning in the subject’s response, providing more relevant results.  
• Limited  set-up time. Not only relevant for the subject’s comfort but also to optimize study timing when a big pool of subjects and consecutive recordings is needed. 

There are different fields in which this research scenario can be applied, some of them being: 

• Screen-based stimulation design. They usually do not require much freedom from the user and, as such, can be performed in more controlled environments (see Figure 4). 
  Said stimuli can be designed to assess different fields, like evaluating emotional reactions in a population with autism. 
• Virtual simulations. They usually demand static dispositions, especially in their first stages of design. Their screen-based displays benefit the most from this option, but it can also be applied in VR or more complex studies.

Figure 5: Multimodal in-lab scenario: Bitbrain Diadem, Bitbrain Ring and Tobii Pro Spark 

              b. Real-Life scenario: 

Unlike the previous case, these studies have the most demanding conditions regarding sensor ergonomics and adaptability.

They usually require fast set-up times and minimalistic system design for a more natural subject experience. They are also usually tied to tests performed under controlled movement, so they also need to have wireless and robust designs.

In these scenarios, some key factors are: 

• Comfortable disposition. Even more relevant than in the previous case, this feature allows the user to have less conditioned behaviour.
• Fastest set-up time. Closely linked and very relevant to achieve the previous factor.
• Dry.  Not having any dependence on liquids or gels allows for the system to be applied in more scenarios without the worry of losing moisture and worsening the signal. 
• Wireless. One of the most relevant factors for movement allowance. 
• Data backup memories. Especially relevant in conditions in which the environment might compromise the connection quality with external systems, or in scenarios in which the researcher wants the subject’s behaviour to not be affected by any close-by recording station. 
• More specialized layouts. Once surpassed in the exploratory phase performed in the laboratory, this optimization allows the researcher to use smaller systems that do not lose information from the key areas of the study.

In this case, some relevant fields of study that might benefit from it are: 

• Studies on environmental impact. Whether it is regarding architectural design, shopfronts, or museum disposition, having as much freedom as possible is important to study the true reactions of the population to these scenarios. 
• Educational environments. The study of the emotional and cognitive responses of kids to educational techniques in classrooms or workshops is a notable example of this application. Not only are ergonomics and speed relevant for the external conditions but also to improve the acceptability from the younger subjects (see Figure 5). 

Figure 6: Multimodal out-lab scenario: Tobii Pro Glasses 3 and Bitbrain Neurohead Band applied for educational research purposes 

       c. Flexible scenarios. 

A wide variety of options will allow the researchers to apply any multimodal combination of these systems to any of their study cases. 
Whether it requires a more specialized layout in a laboratory scenario or a higher spatial resolution in real-life environments, each combination depends on finding a compromise between the features of each one of the sensors. 

Here we will present a chart with key features of different Bitbrain and Tobii systems that have been optimized for these different cases: 

Table 1: Comparison table with technical specifications of Tobii and Bitbrain’s hardware.

5. Research examples 

All these types of laboratories and research scenarios have greatly increased during the last few years. As such, Bitbrain’s and Tobii Pro’s technology has been used by many different researchers all around the world.  
Here are some interesting examples that we would like to share: 

Multimodal behavioral analysis in real-life scenario:  

Impact study on an exhibition at the Maritime Museum of San Sebastián [12] 

This study was performed in 2020 by the Human Factor and user experience (HU&UX) laboratory, from TECNALIA Research and Innovation. 

The contents of the study were “to identify the exhibition’s resources that are the most attractive to the visitors” (Sara Sillaurren, ICT Project Manager and one of the main researchers on this study). As such, the main objective was to “create a final report that determines the key elements of the exposition that evoke the biggest impact on the public, to guide the museum managers on future expositions.” 

To this end, the research required maximum controlled movement allowance and comfort on the subject’s side. The exposition’s environmental conditions required for the tests to extract relevant information on the unconscious response of the subjects to different stimuli: texts, images, videos, and smells. 

The HU&UX research group used:  

• EEG recording – Diadem EEG.  
• GSR and BVP recording – Ring.  
• Eye-tracking recording – Tobii Pro Glasses 3 (wireless).
• Indoor Positioning System – InTrack (not available commercially).  

The multimodal recording was performed with SennsLab, and the data analysis was done with the SennsCloud Platform and SennsMetrics. 

With these tools, the research group extracted a full cognitive and emotional analysis on the public’s response, segmented by age groups. The relevant analysis results were extracted by separating the exhibition into distinct thematic areas and categorizing the stimuli presented on each one of them.

  

Figure 6: Diagram of multimodal out-lab scenario: Tobii Pro Glasses 3, Bitbrain Diadem and Ring. Data flow that specifies the steps followed to carry out the study and obtain the results. 

Multimodal behavioral analysis in-lab static scenario:

Detection of driver’s cognitive states based on LightGMB with multi-source fused data [13]  

This research was made by researchers at Tsinghua University for the WCX SAE World Congress Experience.

Nowadays, it is common for drivers to use mobile phones while driving which leads to cognitive distraction that can cause traffic accidents. The study of the cognitive state of drivers is vital in developing solutions to prevent this increased number of traffic accidents due to mobile phone use.

In order to do this, the researchers focus on studying three aspects:  

1. Vehicle running characteristics measured with ErgoLab software. Based on the literature, they found that there is no significant difference between collecting data from real driving and driving on a simulator. Therefore, the experiment was conducted on a driving simulator, CarSim 2016.
2. Visual feature, for the study of this aspect they used the screen-based Tobii Pro Fusion device. 
3. EEG, for which they used a Versatile EEG 32 ch. 

The LightGMB algorithm broadly assesses whether the measured parameters help to detect the driver's cognitive state to a greater or lesser extent.

The experiment was conducted with 8 participants without any pathology between 20 and 30 years of age, who were made to drive on a two-lane road in the simulator. They were subjected to three different tasks with diverse levels of cognitive load:   

• Task 1: the subject is not distracted and focuses on driving
• Task 2: the subject is called on the phone and asked a series of questions that do not require great concentration as they are conversational. 
• Task 3: the last task consists of telephoning the driver and asking him/her for arithmetic calculations, which involves a state of extreme distraction.

In this case, the multimodal study allows different information to be obtained from the different hardware used.

Measuring vehicle parameters such as the steering angle or the curvature of the road makes it possible to assess the driver's visual distraction. In the case of eye tracking, other parameters such as the speed of gaze movement or the frequency of fixation make it possible to distinguish the subject's cognitive distraction. As well as measuring pupil dilation, which is activated by the sympathetic system in situations that could endanger the organism. Finally, through the study of the EEG signal, the frequencies of the alpha and beta waves can be measured; their increase means that the subject is subjected to a greater cognitive load.

With this multimodal study, this research group managed to analyze the cognitive state of several drivers in different states of concentration, and they also evaluated which parameters provide the most valuable information. 

Figure 7: Diagram of multimodal in-lab scenario: Tobii Pro Fusion, Bitbrain Versatile EEG and Ring. Data flow specifies the steps followed to carry out the study and obtain the results. 

---

Conclusion

The study of human behaviour is a trend of research that is increasingly gaining momentum in fields like human factors, human-machine interaction, sociology, etc.

This post gives a brief overview of how to approach these studies and discusses how researchers believe in the importance of measuring several biological signals allowing to perform a complete study covering more points of view.

In order to carry out these multimodal studies, there are a few things that are important to consider. The first one is the parameters that the researcher wants to study (attention, EEG, GSR, ECG, etc.). This leads to the following question, choose an appropriate technological combination that is correctly synchronized. A last point to bear in mind is that depending on the signals that you want to monitor the hardware that you need will limit the scenario in which you plan to perform these projects. 

This is the biggest limitation of multimodal studies until now, there are factors that force the development of these studies in an in-lab scenario. In these situations, the environmental conditions are controlled, which also has an advantage, as it is possible to perform a study with higher technical specifications. 

However, the development of the latest technologies is focused on facilitating the use of these devices in any situation without losing signal quality. This has made it possible to carry out studies in real-life scenarios where environmental conditions are more demanding.

Finally, this post raises some cases of multimodal studies using a combination of Tobii and Bitbrain technologies in different settings. These investigations were successfully performed, proving that the development of this kind of technology is giving the possibility to carry out studies that are as close to reality as possible in the field of human behaviour.  

6. Bibliography

[1]       Yu, Z., & Wang, Z. (2020). Human Behavior Analysis: Sensing and Understanding. En Springer eBooks. https://doi.org/10.1007/978-981-15-2109-6 
[2]       Mento, Mark A. May 5, 2020. This is How Eye Tracking Technology Works. Bitbrain. 
[3]       Mento, Mark A. June 12, 2020. Different Kinds of Eye Tracking Devices. Bitbrain. 
[4]       Bondi, Marisa. September 3, 2021. How to choose the right eye tracker for your research – the all-in-one guide. Tobii Blog. 
[5]       Shure, Caitlin. February 12, 2021. What is EEG and what is it used for? Bitbrain.  
[6]       Gil-López, Cristina. March 12, 2021. The procedure and Uses of EEG Test. Bitbrain. 
[7]       Niso, G., Romero, E., Moreau, J. T., Araujo, A., & Krol, L. R. (2023). Wireless EEG: A survey of systems and studies. NeuroImage, 269, 119774. https://doi.org/10.1016/j.neuroimage.2022.119774  
[8]       Bitbrain Team. July 14, 2020. The Wet EEG Cap & Differences Between Water-Based, Saline and Gel EEG caps. Bitbrain. 
[9]       Minguez, Javier. May 12, 2020. How to Select a Dry-EEG Headset for your Research Application. Bitbrain.  
[10]     Oliver, Andreu. June 29, 2020. EEG Synchronization With Other Biosensors (EEG, ECG, EMG, eye tracking, etc.), and Software. Bitbrain.  
[11]     Multimodal research solutions - EEG and Biometrics. (s. f.). Tobii Blog. https://www.tobii.com/products/software/behavior-research-software/tobii-pro-lab/multimodal-research-solutions.  
[12]     Olaizola, Borja. January 10, 2020. El “mapa caliente” de Elcano. El diario Vasco. 
[13]      Li, J., Liu, Y., Ji, X., & Tao, S. (2022). Detection of driver’s cognitive states based on LightGBM with Multi-Source fused data. En SAE Technical Paper Series. https://doi.org/10.4271/2022-01-0066