gms | German Medical Science

When questions on a person’s everyday life or on their behavior in certain situations are researched, a high level of ecological validity is desired to ensure the outcome is representative. In an audiological context, ecological validity is defined as “the degree to which research findings reflect real-life hearing-related function, activity, or participation” [1]. Thus a controlled environment is often either impractical or might directly affect the results, since an increase in the level of control often represents a decrease in ecological validity and naturalness [2]. The trusted tools for this sort of research are extensive questionnaires that are usually taken retrospectively, but have been found to be prone to a variety of effects, e.g., memory bias or forgetfulness to name only a few [3]. These limitations motivate the employment of a relatively novel technique named Ecological Momentary Assessment (EMA), which encompasses “real-time collection of data about momentary states, collected in natural environments, with multiple repeated assessments over time” [4].

Currently, EMA studies employ technologies like smartphones to assess the factors of interest in-situ. Short digital questionnaires are typically taken on multiple occasions per day, thus minimizing cognitive effects like, e.g., recall bias, and preserving ecological validity as much as possible. Depending on the sampling strategy of the experiment, factors of interest may be traced as they develop over time. To account for environmental parameters, additional data are often recorded simultaneously [4]. Such parameters might be geographical coordinates, room temperature, or, in an audiological context for example, time-varying hearing-aid settings or acoustical features. In audiology, EMA is used in experiments that relate surrounding acoustical factors to perceptions, feelings, and behavior, e.g., to assess signal-to-noise ratios (SNR) in listening situations [5], or to examine the benefit of hearing aids on the user’s everyday life [6]. Many different tools and combinations have been employed, ranging from recording devices worn around the neck [5] to specialized hearing aids [2]. These are usually supplemented by a smartphone running an assessment application, sometimes inside a web browser, but so far the structural elements have been very specific to the respective study design. There exist commercial solutions for self-assessment, but the recording of objective parameters largely remains each experimenter’s own concern.

This technical note introduces olMEGA, an integrated, open-source toolbox created to perform EMA studies with highly customizable questionnaires, and with built-in acoustical features that are tailored to fully preserve the participants’ privacy. Along with the software package comes an open hardware design for a stereo audio input channel transmitted via Bluetooth. The system is built for Android OS; additional tools were programmed in Matlab. When no objective parameters are needed, the application can also merely be used as a digital survey tool. Figure 1 [Fig. 1] shows the different modules of the system and their interaction with each other.

The olMEGA framework was originally developed for the EMA study “Hören im Alltag Oldenburg (HALLO)” and refined for use in the study “Individual Hearing Aid Benefit in Real Life (IHAB-RL)” [6]. The latter was funded by the Hearing Industry Research Consortium (IRC) and provided insight into the personal benefit of hearing devices in various situations of everyday life. The design of both studies required the system to satisfy two major demands: usability and robustness. To be used by people without technical expertise, it needed to be mostly self-explanatory and highly error-tolerant. It has since become a comprehensive toolbox that offers many degrees of freedom and ease of use for the participants. Researchers can set up a working experiment device in very little time and a detailed step-by-step description is provided in the handbook [7].

The core of the olMEGA framework is the questionnaire tool. It can be used in a stand-alone version without data recording or reminder functionality as a typical, but open-source, easy-to-use, and extensible tool for surveys. One purpose could be the standardized tracking of any arbitrary state or condition such as time-dependent tinnitus pitch [8], but also something as basic as data logging during a customer survey. Different sampling options include scheduled as well as self-initiated questionnaires. Regardless of whether a reminder is imminent, taking a questionnaire is always optional and not mandatory, while surveys can be taken at any given time.

The olMEGA app provides a simple and tidy design (see Figure 2 [Fig. 2]) with clear lines and colors and as little distraction as possible. The user experience was tailored to pose a very low threshold for the participants, as there are only two options to choose from: an element to access a short help section in the top left corner, and a button for taking a questionnaire in the center of the screen. Redundant information is kept from the user, and, if errors occur, a plain and easily understandable message is displayed. Because of the high operational reliability, in most cases this reduces to critical battery level and the associated request to charge the device. Apart from the aforementioned options, some additional information is presented: the current date and time, the battery level, and, if enabled, the remaining time until the next alarm. Once an alarm is set off, the option “Take Questionnaire” in the center of the screen is highlighted while a vibrating alert is active for 30 seconds.

Many EMA studies ask the participant to take questionnaires at a defined interval. Because static intervals may introduce sampling artifacts, e.g., someone is listening to the news every 30 min, a random time difference based on a static time difference plus or minus a certain amount of time can be generated for every new interval. This minimizes the risk of artificially recurring responses while still keeping an equal number of questionnaires completed per day.

The advanced, built-in logic makes each questionnaire an interactive experience of high specificity. Questions may be declared mandatory and compel the participant to give a response. Decision-based branching limits the number of questions presented to what is necessary, without overburdening the participant, basing future questions on previous responses. Decision criteria are easily integrated into the blueprint of every questionnaire and can be scheduled individually for each single question. Multiple response formats are available, e.g., single-choice, multiple-choice, sliders, and more (see Figure 3 [Fig. 3]) so the researcher can tailor each question exactly to the needs of the experiment and the targeted test group. Some questions require the participant to make a choice between several options. In such cases, radio buttons (Figure 3 [Fig. 3], left panel) offer a visual and haptic representation of the task, automatically de-selecting an option whenever a different one is selected by the user. Checkboxes (Figure 3 [Fig. 3], 2^nd panel from left) give the impression of a list, of which multiple alternatives may be selected. If emotional states are to be assessed, a series of emojis (Figure 3 [Fig. 3], 3^rd panel from left) to choose from are a valid and intuitive option and if a question is examining a sensation of level, a slider (Figure 3 [Fig. 3], right panel) could be the appropriate format. The latter provides a free value and a fixed category option, both showing either end points or multiple steps. Free value results range between 0 and 1, rounded to the 8^th decimal place. Navigation through the questionnaire is achieved via swiping or by forward/backward action buttons.

olMEGA enables researchers to implement questionnaires that are tailored to the various needs of any study while taking care of most functionality for them. In fact, the creation of a signature questionnaire is one of the few efforts that experimenters need to undertake themselves, but the process and structure is determined by clear rules. Questionnaires use an adaptation of the .xml format, consisting of tags and values, and a template questionnaire with all types of responses is available together with the software, and may be used as a starting point. A very detailed set of instructions on questionnaire creation is given in the aforementioned handbook [7] and an overview of the design process is presented in Figure 4 [Fig. 4].

Each question may be equipped with a filter parameter, consisting of one or more entries that drives a decision module. The mechanics behind this parameter is that the respective question will only be presented to the study participant when certain criteria are met, e.g., if specific responses to questions were given earlier. Also possible is the opposite, meaning that a question is only presented if certain responses were explicitly not given. Figure 5 [Fig. 5] gives an example of a question design using radio buttons to represent a single-choice option and the resulting image on a smartphone.

The example in [Fig. 5] shows how little code is necessary to build and organize a survey using the olMEGA suite, as only a few elements are needed: besides the obvious <question> and <label>, there are <options>, which represent the response options, and ids, which are user-dependent and serve partially to organize the data. The <text> depicts all that is printed on the screen and if, instead of <option>, a <default> tag is set, the answer is pre-selected by default. In order to lower the entry level even further, a questionnaire designer tool is being developed which will implement drag and drop techniques to facilitate questionnaire generation via a few mouse clicks.

Many EMA studies record data together with the responses from questionnaires [4]. In audiology, this often refers to acoustical features extracted from some arbitrary audio source. The olMEGA framework includes three different stereo-channel features that were selected based on recommendations in [9] as a robust parameter set for rudimentary acoustical scene classification, while, at the same time, only consuming moderate computational resources. The features provided are: RMS to obtain an overall sound level, Zero-Crossing Rate (ZCR), which can be used as a basic pitch estimator and voice activity detector, and (Cross) Power Spectral Densities (PSD), which convey spectral information and coherence measures. With RMS as well as ZCR values being updated every 12.5 ms and PSDs receiving updates every 125 ms, a detailed overview of the participant’s data, called a day print, can be generated.

To accommodate strict data privacy regulations, the spectral information stored in the PSD feature deserves special consideration. Reconstructive measures have proven unsuccessful in recovering meaningful lexical contents such as words and numbers. Studies have found that the selected update rate and a temporal weighting of successive frames with a time constant of 125 ms during the calculation of the feature eliminates the fine structure and thus prohibits any intelligibility [10]. All acoustical features that are provided by olMEGA are therefore fully privacy preserving for speech content, meaning that no context or intelligibility is available.

The software accepts multiple sources of audio input and indicates the chosen source by a specifically colored microphone symbol at the top center of the screen, as shown in Figure 2 [Fig. 2], with a different color for each supported source. Input options, besides the obvious questionnaire-only setting, are the built-in microphone(s), an external audio interface attached via the USB connector of the smartphone, a Bluetooth connection applying the A2DP protocol, as well as two different Bluetooth connections complying with the serial RFCOMM standard. More information on the latter is presented in the next section. Different sampling rates are supported, depending on the signal source, and the feature extraction adapts automatically, so the system can be customized to meet the needs of the study rather than the other way round. Apart from simultaneous use with questionnaires, olMEGA can also be used for general-purpose, hands-off, long-term monitoring of acoustic scenes such as, e.g., for noise-pollution studies or at cultural events, whilst preserving the privacy of all parties. External developers may of course supplement the feature set with their own algorithms.

Several mechanisms have been implemented to protect the participants’ data against unauthorized access. During normal use the mobile phone is governed by the aforementioned kiosk mode, disabling all options to leave or close the application. Furthermore, it ensures that the application is always displayed as the top layer with secondary screens like, e.g., popup windows being directed to the background. In case of a third party trying to download material from the device, a popup request to grant access is displayed on the smartphone but because the front layer is exclusively reserved for olMEGA, no permission can be accorded. Unauthorized access is therefore impossible without intimate knowledge of the system.

Freely available schematics of an optional stereo Bluetooth sound source are included with the olMEGA system [11]. Using this device, acoustic features can be obtained alongside questionnaire results. The hardware was developed in-house and is fully supported by the android application, including automatic re-connection on loss of transmission. It connects to any smartphone or tablet computer, rendering the device an optimal choice for a stereo audio input. The apparatus involves two tethered I2S miniature microphones (see Figure 6 [Fig. 6]) that can, e.g., be attached to the participant’s glasses; it has an approximate battery life of 14 hours. Integrated level calibration and a sampling rate of 16 kHz, together with a dynamic range of more than 60 dB and very low inherent noise, produce a high-quality primary stream for the extraction of meaningful features.

Long-term EMA observations lead to an enormous amount of data. Therefore, data handling and tools to visualize data and find statistically relevant information are very important to enable researchers to find answers to their research questions. olMEGA contains an evaluation toolbox written in Matlab. This toolbox provides convenient means to manage data transfer from the device to a computer, especially for technically inexperienced experimenters, and performs a preliminary analysis of the recorded data (see Figure 7 [Fig. 7]). In the center graph, a technical overview of the experiment is presented. The top half shows various states of the application like, e.g., green for normal operation, blue for charging, or yellow for vibrating alert. In the bottom, Bluetooth activity is displayed, along with a battery readout depicted by the black line. The program also provides basic functionality to operate the smartphone remotely. This is helpful when the device is locked in Kiosk mode. No major programming skills are necessary for this operation. Using this program, subjective as well as objective data can be transferred to and from the database described in the next section.

A shared SQL database has been implemented using Python that combines all subjective and objective data from EMA experiments conducted with olMEGA. The database is searchable and accessible via a web interface or an API, and client programs in Python and Matlab are available. This enables teams that work together to pool and share their information and allows the swift transfer of experiment data between teams. The database is free to set up and use anywhere, and can easily be extended by new features. Using this tool is optional and access rights for third parties are customizable. It may also be used locally as a dedicated filing system for data without the intent of information exchange.

In a first step, all objective data is examined regarding its validity. An availability map (see Figure 8 [Fig. 8]) is established, sharing information on existence and state of the experimental data, as well as questionnaire results over the course of the experiment. The graph points out sections of inferior data and the prevailing issues. In the example, listening effort (LE) extracted from the questionnaires is selected as an item for close observation. Circle colors constitute different activities/situations and the respective listening effort is expressed by the sizes of the circles. Colored x’s mark missing LE ratings.

In order to combine subjective and objective data into one representation, a day print is generated as shown in Figure 9 [Fig. 9]. In this example, the result gives an overview of a workday at the office, with the person commuting to work in the morning between seven and eight o’ clock. Data suggests two half-hour meetings at 10:15 AM and possibly 2:15 PM and a lunch break taken at a restaurant.

Figure 9 [Fig. 9] shows multiple streams of information over a span of approximately eleven hours. In the bottom plot, the two-channel RMS level is presented. Averaged values range between 40 and 80 dB SPL, indicating a moderate loudness distribution with occasional loud events, such as, e.g., noise from driving a car in the morning. Second is the spectral information taken from the PSDs. These indicate varying acoustical scenarios, changes correlating with current situations extracted from the questionnaire responses, which are represented by different colored asterisk symbols, one color for each group of situations. Enclosed with the symbols is a more exact description of the situation. The next plot shows the Normalized Cross-PSD [10], or, more precisely, the real part of the Normalized Cross-PSD averaged between 400–1000 Hz (labeled ‘Coherence’ in the plot). This dimension provides the basis for the automatic communication detection presented in the following and can also be adduced to evaluate the recording’s validity. The top graph of the day print shows two measures, which are estimated offline from the PSD data: Own Voice Sequences (OVS) and Foreign Voice Sequences (FVS). These measures indicate whether the person wearing the microphones was producing speech at the time, or someone in the vicinity was. Based on these measures, conversations may be detected and compared to the responses given in the questionnaires. OVS are also particularly important for estimating the SNR.

Along with the overview graph and day prints, questionnaire-specific descriptive statistics are produced. The two example statistics shown in Figure 10 [Fig. 10] were created to communicate preliminary results to the participants. The top graph shows the fractions of each of the main listening situation categories among all questionnaires. The lower graph presents the frequency of questionnaires containing various listening effort ratings (responses divided into three clusters) for different listening situation categories. Finally, all results are compiled into a pdf handout, combining the objective data with the responses given in the questionnaires, thus creating an overview of the whole EMA experiment for the individual participant.

While many different EMA solutions exist, they mainly only implement either the survey part or the recording of objective parameters and they are often very specialized. olMEGA combines questionnaires and the extraction of acoustical features in a single application that is easy to handle for both researchers and participants. The structure of the acoustical features preserves the privacy of participants and bystanders. All data is stored locally on the device with no internet connection needed and, therefore, no third parties involved. Although browser-based applications have the advantage of being platform-independent, olMEGA offers a fully supported Kiosk mode that participants are unable to exit. This contributes to the operational reliability of the app and also reassures technically inexperienced participants. All software, as well as hardware schematics and a user handbook, are freely available under [7], and new features are still being added. We cordially invite developers to contribute to the olMEGA framework

The authors declare that they have no competing interests.

This work was funded by the Hearing Industry Research Consortium (IRC). Additional financial support was granted by the governmental funding initiative “Niedersächsisches Vorab” of the Lower Saxony Ministry for Science and Culture, research focus “Hören im Alltag Oldenburg (HALLO)”. We wish to thank Alejandro Garavito Arango for data handling and all others who participated in this project for their contribution. English language services were provided by http://stels-ol.de/.