Non‐native english language speakers benefit most from the use of lecture capture in medical school

Medical education in the United States and Canada continues to evolve. However, many of the changes in pedagogy are being made without appropriate evaluation. Here, we attempt to evaluate the effectiveness of lecture capture technology as a learning tool in Podiatric medical education. In this pilot project, student performance in an inaugural lecture capture‐supported biochemistry course was compared to that in the previous academic year. To examine the impact of online lecture podcasts on student performance a within‐subjects design was implemented, a two way ANCOVA with repeated measures. The use of lecture capture‐supported pedagogy resulted in significantly higher student test scores, than achieved historically using traditional pedagogy. The overall course performance using this lecture capture‐supported pedagogy was almost 6% higher than in the previous year. Non‐native English language speakers benefitted more significantly from the lecture capture‐supported pedagogy than native English language speakers, since their performance improved by 10.0 points. Given that underrepresented minority (URM) students, whose native language is not English, makes up a growing proportion of medical school matriculates, these observations support the use of lecture capture technology in other courses. Furthermore, this technology may also be used as part of an academic enrichment plan to improve performance on the American Podiatric Medical Licensing Examination, reduce the attrition of URM students and potentially address the predicted minority physician shortage in 2020. BIOCHEMISTRY AND MOLECULAR BIOLOGY EDUCATION Vol. 39, No. 6, pp. 416–420, 2011

Medical education around the United States and Canada continues to evolve as it struggles to meet the projected physician shortage in 2020 [1]. This projected physician shortage may become a reality, even though, according to a recent publication from the Association of American Medical Colleges [2], over the last three decades there has been an increase in the number of underrepresented minority (URM) students attending medical school, and URMs make up a growing proportion of medical school matriculates. The Council on Graduate Medical Education [3] defines URMs as racial and ethnic populations who are represented in lower proportions in the health professions relative to their percentage in the US population as a whole. This definition includes African Americans, American Indian or Alaska Natives, Native Hawaiians or other Pacific Islanders, Hispanic or Latino. If URM students are to contribute significantly to addressing this projected physician shortage, additional strategies are needed after their matriculation to aid their retention in medical training [4,5], since URMs are at increased risk of attrition [6]. Pedagogic changes that enhance the performance of URM students, particularly, those whose native language is not English should be encouraged. Only in this way, will the need for more minority physicians, and a more diversified and culturally sensitive physician workforce be satisfied. Although defining native speakers of English is complex, for the purpose of this study, a person was considered to be a native speaker of English, if English was the language acquired and learned in early childhood, and was the primary language spoken in the family; though there are exceptions [7].
We have previously [8] reviewed the many recent technological innovations in medical education implemented as a result of the report by the Association of American Medical Colleges ''Educating Doctors to Provide High Quality Medical Care [9].'' At the time, we cautioned about the indiscriminate use of technology ''because it's there'' or ''because everyone else is using it.'' We also advocated for the appropriate use of technology only when it has been evaluated and tested and represented an improvement over current practices. The present study represents a step toward that needed evaluation, and its timing seems appropriate a century after the submission of the Flexner report to the Carnegie Foundation [10].
A recent Wainhouse Research White Paper reports that the latest technological innovation sweeping through medical schools is Lecture Capture [11]; defined as any technology that allows the professor to record lecture material and make it available digitally to the student. Although the use of lecture capture technology at medical schools across the United States is becoming increasingly ubiquitous [12][13][14], there remains a paucity of quantitative information on the use of lecture capture technology by medical students and particularly on the impact of this technology on URM student learning.
In this pilot project, 60 first-year students of Podiatric Medicine were asynchronously allowed to access podcasts of lectures captured using the Articulate Presenter software through a password-protected Blackboard learning environment. In this context, podcasts are defined as enhanced online audio presentations where lecture slides are accompanied by faculty narration. These podcasts were not meant to contain additional material, but rather represent the lecture in an alternate format to satisfy the needs of a diverse student body. Although this format is more commonly used in a distance education paradigm, in this study it was used to complement the traditional campus-based face-to-face instruction of a biochemistry course in the Podiatric Medicine curriculum at Barry University, a federally designated minority serving institution with a Carnegie classification of Doctoral/Research University. The Articulate Presenter software has a searchable feature that allows students to study more efficiently by searching each online lecture for a ''key term.'' This feature reveals only those lecture slides (and associated audio) on which that ''key term'' appears. Furthermore, students may ''pause'' or ''speed-up'' the lecture podcast depending on their specific needs. The student therefore assumes control of content delivery. Each lecture podcast was recorded by the instructor before the actual class time and was available online immediately after class. Supplemental course materials were also available to students and included a course note package that was essentially a script of the audio narration. In this way, the online course content could be delivered effectively to the students that were hearing-impaired, and those students whose native language was not English. Thus, the biochemistry course described here is a blended or hybrid course, delivered using a pedagogically sound combination of online and face-face lectures, along with supplemental learning resources and online quizzes that offers the student maximum flexibility.
The aim of this study was to appropriately evaluate the impact of this lecture capture technology on student performance in an authentic learning environment and investigate the extent to which its impact differs for those students who are non-native English speakers.

Study Design and Analysis
All students enrolled in the first year of the Podiatric Medicine program are registered to take the biochemistry course in their first semester; thus control and intervention groups were ''recruited'' in exactly the same way. IRB approval was received to use archival data, so students never gave explicit consent to participate in this study and consequently no students declined participation. Online lecture podcasts were made available to all registered students through a secure server for the first time in Fall 2009. Students were able to view each lecture podcast an unlimited number of times, though they could not download or save them to portable devices.
Student performance in this inaugural lecture capture-supported course was compared to that in the previous academic year (2008). The biochemistry course taught in both 2008 and 2009 were identical in terms of faculty, assessment strategy and all course materials (including the course text that served as a lecture transcript in the 2009 iteration), other than the availability of lecture podcasts made available through the Blackboard learning environment in the 2009 course iteration. Both courses were evaluated using three-unit tests and a noncumulative final examination taken in finals week (50 points on each examination). The examinations for both cohorts were identical, and the examinations were not returned to the students following administration in an effort to maintain the integrity of the examinations. To examine the impact of these online lecture podcasts on student performance a withinsubjects design was implemented, a two-way ANCOVA with repeated measures.
Individual differences are usually part of the error term, increasing the error term and decreasing the power of hypothesis tests. ANCOVA with repeated measures controls for some of the individual variation. This is often called using subjects as their own controls. Note that since students were not randomly assigned to pedagogy, the groups are not comparable in terms of confounding variables such as educational background and motivation; consequently, a covariate (MCAT biology score) was used to statistically estimate student performance ''as if'' every student had an identical biology MCAT score. Thus, the use of a covariate simulates a true experimental design in which all participants have identical educational background and motivation.

Demographics
There were no statistically significant differences between the students in the control (N ¼ 53) and intervention (N ¼ 60) groups in terms of average age, ethnicity, or gender (Table I). The average age was 25.4 6 4.5 years. Sixty two percent were men and 35% were ethnic minorities. Furthermore, there was no statistically significant difference in the ethnicity distribution (Table I) between the groups.

Assessment of the Reliability and Validity of the Course Examinations
Reliability is a measure of the extent to which a test produces consistent results, whereas validity is a measure of the extent to which a test actually measures the knowledge of the subject matter. Reliability is a necessary, but not sufficient, condition for validity. Based on Cronbach's alpha (a ¼ 0.86), the reliability of the four examinations was satisfactory. Criterion-related validity refers to the degree to which the examination scores are correlated with the scores on a second measure (the criterion) that has already been established as a valid measure of the construct. Criterion-related validity was established by correlating the scores on the examinations with biology MCAT scores, see Table II. All test scores are correlated with the MCAT biology score and with each other, p < 0.001.

The Effect of Pedagogy and Native Language on Student Achievement
A two-way analysis of covariance with repeated measures was conducted to evaluate the effect of the lecture capture-supported pedagogy on student achievement, after controlling for prior knowledge and native language. The dependent variable was the score on the four instructor-designed examinations. The within-subjects factors were pedagogy with two levels (traditional and lecture capture-supported), native language with two levels (English native language and other native language), and time of testing with four levels (3, 6, 10, and final week [week 15]). The potential confounding variables, native language, and prior knowledge, were statistically controlled by using native language as a factor and using the MCAT biology score as a covariate. The interaction between pedagogy and test performance at each time was tested using the multivariate criterion of Wilks' lambda (L). The interaction effect was significant, indicating that test performance over time was in fact influenced by pedagogy, L ¼ 0.84, F (3, 64) ¼ 4.19, p ¼ 0.009. Students in the intervention group obtained higher test scores averaged across the four tests (M ¼ 42.3, SE ¼ 1.4) than the students in the control group (M ¼ 36.6, SE ¼ 1.4), p ¼ 0.006. On average the impact of the lecture capture-supported pedagogy was large, improving test performance by nearly a full standard deviation, d ¼ 0.95. Additionally, the interaction between native language, pedagogy, and test performance over time was also significant, indicating that the impact of pedagogy on test performance depends on native language, L ¼ 0.84, F (3, 64) ¼ 3.96, p ¼ 0.012. As shown in Table III, for non-native English language speakers the lecture capture-supported pedagogy improved test performance by an average of 10 points (43.9 vs. 33.9), whereas for the native English language speaker it showed no significant improvement (40.1 vs. 39.4).
The main effect of gender and the interaction between gender, pedagogy, and test performance at each time was also tested. The main effect was not significant, F (1, 68) ¼ 0.971, p ¼ 0.33. The interaction effect was also not significant, indicating that the effect of lecture capture was the same for men and women, L ¼ 0.994, F (3, 64) ¼ 0.134, p ¼ 0.94.
From Figure 1 we see that on all tests, non-native English language speakers benefitted more from the change in pedagogy than native English speakers, as measured by the gain in test scores from the lecture capture-supported pedagogy. As the raw test scores have no meaningful interpretation, gain scores were standardized, meaning they were converted from raw scores to units of standard deviations. The advantage that the lecture capture-supported pedagogy gave to non-native English language speakers at the final examination in week 15 was especially striking. Compared to the traditional pedagogy, non-native speakers scored more than three and a half standard deviations higher with the use of the lecture capture-supported pedagogy on the final examination.
Considering only the final examination, the difference between the control and intervention groups was significant in a one-tail test for both native English language speakers, p ¼ 0.03 and for non-native speakers of English p < 0.001. However, the effect size as measured by the standardized difference in means was small, d ¼ 0.13 for native speakers and very large for non-native speakers, d ¼ 1.67.  FIG. 1. Standardized difference between lecture capture-supported and traditional pedagogy by native language. DISCUSSION A small but growing number of Medical School faculty continues to innovate in the delivery of course materials to an increasingly diverse group of students [8], many of whom are non-native English language speakers. In the 21st Century, medical education is changing, as more emphasis is placed on the use of technology [9]. Students are embracing this paradigm shift; they even demand it [15], driving the rapid implementation of the latest technologies in medical education, often without appropriate evaluation of their effectiveness.
We report here that lecture capture technology, when used to supplement traditional face-to-face instruction, really does improves student performance in a blended biochemistry course, with non-native English language speakers benefitting more than native English language speakers.
Furthermore, there was no difference in average performance between men and women and both genders benefitted equally from the use of this technology, since no gender differences were observed. All students in this course accessed the podcasts of lectures, often repeatedly, particularly around examination time, and reported high levels of satisfaction with the learning environment. An observation supported by the many positive comments on the ''end of semester''; teaching evaluation surveys, by tracking student's use of the various components of the Blackboard learning environment and by others [16]. Contrary to the findings of McNulty et al. [17] and Grabe and Christophersen [18], who reported only limited use of online lecture podcasts by medical students, our students accessed the online lecture podcasts repeatedly. We were however not able to establish how our students utilized the online podcasts, the degree to which they used the ''pause'' and ''search'' features and what features of the lecture-capture platform they found most useful. This is perhaps a limitation of our study, particularly since there is evidence [19] to suggest that student performance is correlated with the degree to which they interact with the online environment. It has been previously shown by others that while listening to podcasts repeatedly improves student performance, particularly when accompanied with note taking [20], contradictory evidence does exist [18].
Additional limitations to this study may also exist. First, some potentially confounding variables, such as student ''personal issues'' were not controlled, so estimates of the effect of lecture-capture technology may be biased. Second, since a convenience sample was used, caution must be exercised in extrapolating these results to other courses and other institutions. Third, since the intervention was administered by only one individual, it is not assured that the intervention can be effectively administered by other instructors. Furthermore, though the use of ANCOVA in this study is justified, since there are no fundamental demographic differences between the control and intervention groups even though they were from two different cohorts of students, the design of this study would be improved had the same cohort of students been exposed to both pedagogies at different times.
The combination of learning resources reported in this study resulted in an overall course performance that was almost 6% higher than in the previous year. The improvement in student learning reported in this study may be explained by the ability of students to ''pause'' the online podcasts and review the online materials more slowly. This, so-called ''innovative advantage,'' (implicitly defined by the caption of Fig. 1 as the standardized difference between lecture capture-supported and traditional pedagogy) due to the change in the pedagogy was greater for non-native English language speakers in all examinations, though it was particularly evident on the final examination ( Fig. 1) when good time management skills are needed, since there is less time to study between the examinations in final week.
To our knowledge, this is the first study to actually quantify an improvement in examination performance for URM students who are non-native English language speakers in a lecture-capture supported course, though others have previously reported increased levels of student satisfaction amongst URMs when online resources are made available [16,21]. This should not be surprising, since the availability of online resources allows the modern day medical student, with their increased personal and social demands [22] to access lecture content at their convenience. It provides an opportunity for students to augment lecture attendance with review of difficult concepts online repeatedly, catch up on material if a lecture is skipped, and it provides an alternate lecture resource for those who do not learn well in large lecture theaters.
Although medical students generally report high levels of satisfaction with these supplemental online resources, it has been reported that too much flexibility can adversely effect student learning [23], as students miss class to ''cram'' for upcoming examinations, intending to review the skipped lecture online; they rarely do. Furthermore, when left to their own devices, some students may utilize the online lectures in a manner that does not benefit learning [24]. There are also some perceived disadvantages to the widespread use of lecture capture in medical education expressed by faculty. For example, faculty often expresses concern that this technology will adversely affect lecture attendance. These concerns however are unfounded as reported previously [17,25,26]. Although collecting attendance data can be cumbersome in large classes [18], and no attendance records were kept in this study, it appeared to faculty that class attendance was equivalent to that in previous years. Additionally, faculty is often concerned that the use of this technology may detract from the holistic medical school experience, specifically the interaction between students and faculty, as well as diminishing class cohesion. These concerns could easily be addressed by the implementation of class tutorials or seminars to discuss clinical correlations, for example. One further disadvantage, at least for this faculty member, that should also be addressed, is the extremely labor intensive process of podcast preparation. In this study, it took faculty 4 hr to generate the audio and slide-building animations to accompany the PowerPoint slides for 1 hr of lecture. However, podcasts captured in this way were very ''clean,'' devoid of extraneous sounds (students arriving late, background conversation, cell phones) that could potentially distract students from content delivery, and the audio recording was of consistently high quality.
In addition to improving student performance in this blended biochemistry course, lecture capture technology has many other applications. We envisage that the availability of lecture podcasts in other courses could result in similar improvements in student learning. Use of this technology could be effectively utilized in a variety of remediation strategies and aid in student retention, particularly the retention of URM students. Furthermore, use of this technology in some sort of Board review course could improve American Podiatric Medical Licensing Examination scores, particularly for students whose native language is not English. This is significant, since increasing the number of URM physicians is a long-term goal of the American Medical Association. The use of these online podcasts throughout the medical curriculum may also increase institutional competitiveness as students seek out institutions offering a variety of innovative learning resources.
Future studies might usefully explore how medical students use these online resources, the strategies they find most effective and what features of the lecture capture platform they employ most frequently. Furthermore, it is interesting to speculate on how student learning would be affected, should the online resources be made available to students in advance of class time, allowing for more interaction, exploration, and collaboration in the classroom. Of particular importance is the implementation of further studies to validate the effectiveness of lecture capture technology as an educational tool before it is more widely adopted.