gms | German Medical Science

GMS Journal for Medical Education

Gesellschaft für Medizinische Ausbildung (GMA)

ISSN 2366-5017

OnlineTED.com − a novel web-based audience response system for higher education. A pilot study to evaluate user acceptance

project medicine

Search Medline for

  • Felizian Kühbeck - Technische Universität München (TUM), Institute of Pharmacology and Toxicology, Munich, Germany
  • Stefan Engelhardt - Technische Universität München (TUM), Institute of Pharmacology and Toxicology, Munich, Germany
  • corresponding author Antonio Sarikas - Technische Universität München (TUM), Institute of Pharmacology and Toxicology, Munich, Germany

GMS Z Med Ausbild 2014;31(1):Doc5

doi: 10.3205/zma000897, urn:nbn:de:0183-zma0008978

This is the English version of the article.
The German version can be found at: http://www.egms.de/de/journals/zma/2014-31/zma000897.shtml

Received: July 2, 2013
Revised: October 15, 2013
Accepted: November 20, 2013
Published: February 17, 2014

© 2014 Kühbeck et al.
This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by-nc-nd/3.0/deed.en). You are free: to Share – to copy, distribute and transmit the work, provided the original author and source are credited.


Abstract

Background and aim: Audience response (AR) systems are increasingly used in undergraduate medical education. However, high costs and complexity of conventional AR systems often limit their use. Here we present a novel AR system that is platform independent and does not require hardware clickers or additional software to be installed.

Methods and results: “OnlineTED” was developed at Technische Universität München (TUM) based on Hypertext Preprocessor (PHP) with a My Structured Query Language (MySQL)-database as server- and Javascript as client-side programming languages. “OnlineTED” enables lecturers to create and manage question sets online and start polls in-class via a web-browser. Students can participate in the polls with any internet-enabled device (smartphones, tablet-PCs or laptops). A paper-based survey was conducted with undergraduate medical students and lecturers at TUM to compare "OnlineTED" with conventional AR systems using clickers. "OnlineTED" received above-average evaluation results by both students and lecturers at TUM and was seen on par or superior to conventional AR systems. The survey results indicated that up to 80% of students at TUM own an internet-enabled device (smartphone or tablet-PC) for participation in web-based AR technologies.

Summary and Conclusion: “OnlineTED” is a novel web-based and platform-independent AR system for higher education that was well received by students and lecturers. As a non-commercial alternative to conventional AR systems it may foster interactive teaching in undergraduate education, in particular with large audiences.

Keywords: audience response system, interactive teaching


Introduction

There is an increasing shift in higher education from teacher-centered to more interactive, student-centered teaching concepts. This is in part due to the known limitations of traditional lectures, such as low student attention or inefficient knowledge acquisition and translation [1]. To foster interactive learning and student engagement, in particular with large student numbers, audience response (AR) systems are increasingly used at educational institutions worldwide [2]. AR systems typically consist of an hardware transmitter (“clicker”) that is controlled by the student, a radio-frequency receiver and a computer with software to display voting results during the presentation by the instructor, e.g. as histogram of responses [3]. AR systems are used in a variety of ways: to increase student interaction and attention, to promote in-class discussions, to evaluate student knowledge or as formative assessments for the lecturer to adjust teaching pace and didactics [2]. Recent meta-analyses in health profession education showed that AR systems are favored by the vast majority of students and likely result in improved short-term and long-term knowledge outcomes [4]. However, many instructors only reluctantly use AR systems in teaching. Frequently cited reasons are high acquisition and maintenance costs, logistic efforts or technical difficulties to operate AR system with different operating systems and presenter software [5].

To overcome these limitations we sought to develop a novel web-based AR system that is specifically adapted to the didactic needs and economic requirements of higher education. Here we present "OnlineTED", a novel smartphone- and web-based AR system that is free-to-use, platform independent and does not require hardware clicker or additional software. Based on user evaluation data, we analyzed the strengths and limitations of “OnlineTED” in comparison with a conventional AR system that is used at TUM and discuss the potential of web-based AR systems for undergraduate medical education.


Methods

Programming and online database

“OnlineTED” was developed and programmed by students and lecturers at TUM. Hypertext Preprocessor (PHP) as server-side programming language was used to build a web-based AR system that is compatible with different operating systems. The user interface was linked to a My Structured Query Language (MySQL) database for storage and management of user accounts and data. JavaScript was used as client-side programming language for animated features, e.g. a poll countdown function. Hypertext Transfer Protocol Secure (HTTPS) was implemented to ensure secure internet communication for the user.

Structure and features

The AR system “OnlineTED” can be accessed via the open-access website http://www.OnlineTED.com (German version: http://www.OnlineTED.de). After log-on, the main page conveys information for the auditorium to participate in a poll (e.g. website URL for login, session token, quick response (QR) code). For the lecturer, the main page is the starting point to create, manage or commence polls. Flags link to the sub-pages "poll viewer" to start of a polling session, "quick poll" for spontaneous polls and "poll editor" to create or manage question sets (see Figure 1 [Fig. 1]). In the latter, new poll sets and questions in multiple-choice format can be created and saved via an online form (see Figure 2 [Fig. 2]). In addition, image (gif, jpg or png), movie (mov, avi or mpeg) or sound files (mp3, aac or wav) can be uploaded and saved with each question. Other media sources of the internet (e.g. http://www.youtube.com/) can be directly embedded. In addition to prepared and saved questions sets, the lecturer can conduct spontaneous polls by choosing different rating scales (yes / no / undecided; very low - very high; excellent - insufficient; strongly agree - strongly disagree) from the "quick poll" options.

Students can participate in the poll after log-on to the website with any internet-enabled device (smartphone, tablet-PC, laptop). A poll is joined by entering a four-digit access code (session token) that is automatically generated and displayed for each session. Alternatively, students can scan a QR code with their mobile device. Upon session log-on, answer options are automatically displayed on the student devices (see Figure 3A [Fig. 3]). After voting, the results are dynamically displayed in the presentation by the lecturer as growing histograms depicting the number and percentage of votes for each answer (see Figure 3B [Fig. 3]).

Each poll question requires a bidirectional data transfer of approximately 0.03 Mbit. Based on the average connections speed of wireless Local Area Network (LAN) (54 MBits/s) or cellular mobile networks such as Universal Mobile Telecommunications System (UMTS) (7.2 Mbit/s) [6], “OnlineTED” enables polls with large audiences in a typical lecture hall setting.

The AR system was programmed in PHP as a server-sided programming language to be compatible with all operating systems without the requirement to install additional software.

Study design and user evaluation

To evaluate the use of “OnlineTED” in higher education, the tool was tested with undergraduate medical students and instructors in lectures (average number of participants: 200±50) and seminars (average number of participants: 22±3) of pharmacology at TUM in the winter term of 2012/13. A conventional AR system using hardware clickers (Interwrite Cricket , Scotsdale, USA) that is routinely used for teaching at TUM served as a comparison.

Information on usability and user perception of the two different AR systems was solicited through a paper survey at the end of the term. A 1-to-6 rating scale was used to assess the overall rating of "OnlineTED". The answer options were 1 "excellent", 2 "very good", 3 "good", 4 "fair", 5 "poor" and 6 "inadequat". For the comparison of "OnlineTED" and the conventional AR systems, survey participants were asked to select the option "superior", "equal" or "inferior" for each of the categories, respectively.

Study participation was voluntary and both informed consent by the participants and approval by the ethics committee of TUM were obtained. Website statistics were tracked by Google Analytics (Google, Mountain View, CA).


Results

Comparison “OnlineTED” vs. conventional AR system

A conventional AR system using hardware clickers (Interwrite Cricket , Scotsdale, USA) that is routinely used for teaching at TUM served as a comparison. Interwrite Cricket consists of 2.4 Ghz band radio receiver and radio transmitters (clicker). The system can manage up to 2047 clickers per receiver. The keypad of the clicker consists of six numeric and additional true and false buttons. The clickers are powered by two AAA batteries each that last approximately 20 weeks based on typical usage. The operative range of the clickers is up to 150 feet (45.7 m) from the receivers (in free space). Interwrite Cricket software runs on Windows, Mac and Linux and supports the generation of multiple-choice and true/false questions with a authoring software that requires installation on the computer of the presenter (http://www.banxia.com/prs/hardware/cricket).

In contrast, "OnlineTED" is a web-based AR system that utilizes internet-enabled end devices of the audience as handsets. "OnlineTED" does not require the installation of software, is fully web-based and thus can be used on any operating system. Each poll question requires a bi-directional data transfer of approximately 0.03 Mbit, which enables the participation of large audiences even with low data bandwidth. The operative range of "OnlineTED" is not spatially limited and thus can be used for distance teaching, e.g. webinars. Sets of multiple-choice questions can be created and manage in "OnlineTED", and spontaneous polls with different rating scales (yes / no / undecided; very low - very high; excellent - insufficient; strongly agree - strongly disagree) conducted as "quick polls" options.

Survey participation

To evaluate "OnlineTED" and compare its user acceptance with conventional AR systems, a paper-based survey was conducted with students and lecturers at TUM. 361 of 709 (51%) students participated in the survey. Of these, 203 students were in their first clinical year (third year of study) at TUM and enrolled in the course "basic pharmacology" consisting of lectures and seminars. 158 students were in the third clinical year (sixth year of study) at TUM and enrolled in the course "clinical pharmacology" that consisted of lectures only. The average age of the students enrolled was 23.4 (SD 3.5) years in the course "basic pharmacology" and 25.3 (SD 3.3) years in the course "clinical pharmacology". Of the lecturers, nine of ten participated in the survey. Their average age was 40.1 (SD 10.0) years.

Web statistics and evaluation data

The beta-version of "OnlineTED" was made available on November 1st, 2012 at http://www.OnlineTED.com (German version: http://www.OnlineTED.de). As of February 28th, 2013, the website has been accessed by 2,997 visitors with 37,696 page hits (approximately 750 visitors and 9,424 page hits per month).

The overall rating of “OnlineTED” by lecturers was "very good" (5 out of 10) or "excellent" (4 out of 10). The didactic benefit of “OnlineTED” for teaching was considered "excellent" by 6 out of 10 and "very good" by 3 out of 10 lecturers. Free text comments by the lecturers indicated that “OnlineTED” provided a "helpful basis for in-class discussions between students and lecturer”. In addition, "playful discussion of topics and clinical problems", e.g. by presenting clinical cases and questions on pharmacotherapy, drug choice and clinical management were considered a key didactic benefit of AR systems. A specific advantage of "OnlineTED" was seen in the possibility to "spontaneously assess the learning and knowledge state" of the students by using the instant poll option.

When asked to compare “OnlineTED” with conventional AR systems using hardware clickers, the lecturers rated “OnlineTED” superior in seven out of nine categories (see Figure 4A [Fig. 4]). The areas with the highest advantage for lecturers were seen in time required to use the TED system for the first time, well as its flexibility (10 out of 10 and 9 out of 10, respectively).

Approximately 80% (289 of 361) of the undergraduate medical students in the studied cohort at TUM indicated to own a mobile internet-enabled device (smartphone, tablet-PC or iPad). Of these, the majority (92%, n=265) could access the polling session without difficulty using either wireless LAN or cellular data network during lectures or seminars. The vast majority of students (94%, n= 274) gave an overall rating for “OnlineTED” of "excellent" or "very good". “OnlineTED” was seen on par or superior in all categories when compared to conventional AR systems using hardware clickers (see Figure 4B [Fig. 4]).


Discussion

In this article we present “OnlineTED”, a novel web-based audience response system that is free-to-use, platform-independent and does not require hardware clickers. Our aim was to develop an AR system for higher education that is on par with commercial systems with respect to functionality and usability. Surprisingly, “OnlineTED” was evaluated superior in a number of categories by both students and lecturers (see Figure 4 [Fig. 4]). In particular, its usability and flexibility with regard to handling, question management and in-class performance were seen as key advantages of "OnlineTED" in comparison to conventional AR systems. Finally, "OnlineTED" is a non-commerical AR system that is freely available to educators and students worldwide.

A potential limitation of "OnlineTED" is its dependency on internet availability that may be limited or restricted at some university campuses due to poor WLAN coverage or cellular network reception. Another important issue is the safety of data on the internet. To address these concerns, “OnlineTED” has implemented HTTPS, a communications protocol for secure communication over a computer network that is used for sensitive data transfer e.g. electronic banking. HTTPS provides bi-directional encryption of data transfer between a client and server and authentication of the web site and associated web server [7]. As an additional safety measure, “OnlineTED” does not permit users to identify the IP address of devices used in a poll, thus assuring the anonymity of poll participants. Nevertheless, further studies are needed to investigate the concerns and attitudes of students and lecturers to web-based technology and its impact for higher education.

Extensive research on AR systems in higher education has indicated their effectiveness on different learning outcomes, such as knowledge gain, student self-confidence and positive learner reactions (reviewed by [5], [8], [9]). The most commonly cited benefit of AR systems are the increased student engagement through active participation. However, while AR systems provide a convenient way to improve student interactivity, it is in the end pedagogy and didactic implementation that matters. In a recent study by Smith et al. [10], students were encouraged to discuss TED clicker questions with their classmates prior to voting. The authors demonstrated that peer discussions are effective means of active learning that resulted in significantly improved conceptual understanding when used with AR systems. In this regard, the frequently cited limitation of AR systems that not all students may be able to participate in a poll due to limited numbers of clickers or internet-enabled devices may offer the opportunity for collaborative voting and peer discussions in class. Another pedagogical aspect of AR systems, which is often overlooked, is its use in formative assessment and feedback. A recent report by Alexander et al. showed that AR systems can be successfully used to monitor the progress of students in the curriculum and allow students to compare their learning progress with other classmates [11] .

The omnipresence of the internet and availability of computer-based technologies open up new opportunities to actively engage students in teaching. Our survey revealed that up to 80% of undergraduate medical students at TUM posses an internet-enabled device (smartphone, tablet or iPad). This results is in accordance with recent studies that 60 to 80% of UK medical students own a smartphone [12], [13]. Web-based AR systems such as “OnlineTED” may thus provide a feasible alternative to conventional, hardware-based AR technology in teaching and become a suitable tool to foster interactive learning and student engagement in undergraduate medical education, in particular with large audiences.


Acknowledgements

We thank Leszek Wojnowski for critical reading of the manuscript. We are grateful to Pascal Berberat and co-workers for providing statistical data on the age distribution of the survey cohort.


Competing interests

The authors declare that they have no competing interests.


References

1.
Biggs J, Tang C. Teaching for quality learning at university. Maidenhead (Berkshire): Open University Press; 2011.
2.
Kay RH, LeSage A. A strategic assessment of audience response systems used in higher education. Aus J Educ Technol. 2009;25(2):235-249.
3.
Barber M, Njus D. Clicker evolution: seeking intelligent design. Cell Biol Educ. 2007;6(1):1–8. DOI: 10.1187/cbe.06-12-0206 External link
4.
Cate Ten O, Durning S. Peer teaching in medical education: twelve reasons to move from theory to practice. Med Teach. 2007;29(6):591–599. DOI: 10.1080/01421590701606799 External link
5.
Caldwell JE. Clickers in the Large Classroom: Current Research and Best-Practice Tips. Cell Biol Educ. 2007;6(1):9–20. DOI: 10.1187/cbe.06-12-0205 External link
6.
Schüring M. Mobile cloud computing–open issues and solutions. Enschede: University of Twente; 2011. Zugänglich unter/available from: http://referaat.cs.utwente.nl/conference/15/paper/7247/mobile-cloud-computing-open-issues-and-solutions.pdf External link
7.
Callegati F, Cerroni W, Ramilli M. Man-in-the-Middle Attack to the HTTPS Protocol. IEEE Security & Privacy. 2009.
8.
Boscardin C, Penuel W. Exploring benefits of audience-response systems on learning: a review of the literature. Acad Psych. 2012;36(5):401–407. DOI: 10.1176/appi.ap.10080110 External link
9.
Nelson C, Hartling L, Campbell S, Oswald AE. The effects of audience response systems on learning outcomes in health professions education. A BEME systematic review: BEME Guide No. 21. Med Teach. 2012;34(6):e386–405. DOI: 10.3109/0142159X.2012.680938 External link
10.
Smith MK, Wood WB, Adams WK, Wieman C, Knight JK, Guild N, et al. Why peer discussion improves student performance on in-class concept questions. Science. 2009;323(5910):122–124. DOI: 10.1126/science.1165919 External link
11.
Alexander CJ, Crescini WM, Juskewitch JE, Lachman N, Pawlina W. Assessing the integration of audience response system technology in teaching of anatomical sciences. Anat Sci Educ. 2009;2(4):160–166. DOI: 10.1002/ase.99 External link
12.
Robinson T, Cronin T, Ibrahim H, Jinks M, Molitor T, Newman J, Shapiro J. Smartphone use and acceptability among clinical medical students: a questionnaire-based study. J Med Syst. 2013;37(3):9936. DOI: 10.1007/s10916-013-9936-5 External link
13.
Payne KFB, Wharrad H, Watts K. Smartphone and medical related App use among medical students and junior doctors in the United Kingdom (UK): a regional survey. BMC Med Inform Decis Mak. 2012;12:121. DOI: 10.1186/1472-6947-12-121 External link