gms | German Medical Science

How am I supposed to know if I am qualified to study dentistry? This question arises for both applicants and faculty officials, who are keen to keep their student drop-out rates as low as possible. One answer to this question can be found in Heldmann’s definition of a person‘s aptitude for study: “Aptitude for study includes not only specific intellectual inclinations, prior academic education and instrumental conditions, but also a quality that can best be described as a desire to cultivate one’s formation of the heart.” [1], p. 184, [translation by author].

Just like the “formation of the heart” other facets of study aptitude like strong interest in the subject, curiosity and communications skills are difficult to measure [2]. Therefore, study success is commonly used as the outcome measure. Unlike aptitude for study, study success can be operationalized in a number of different ways – from results in interim and final examinations or in individual seminars, to length of study, number of repeat examinations and personal satisfaction with the study course. Examination results are useful as a standard for quantification of study performance on a clearly defined scale and are based on data that are easy to collect [3].

Average school grade points (GPA) are generally considered to be the best predictors of study success measured in grades [4]. The meta-analysis published by Trapmann et al. [5] in 2007 contains all available studies in the European academic region since 1980 on the correlation between school grades and study success. The authors reported mean, corrected validity coefficients ranging from ρ=0.26 to ρ=0.53 for study results, whereby the peak prognostic validity is attained by GPA acquired in Germany. However, it’s questionable to use the German GPA as a general indicator of study aptitude and as a selection criterion, because of its different regulations in the federal states of Germany [6].

Specific results for dental study courses have been reported by Arnold et al. (2011) [7]. The authors found moderate, positive correlations between the German GPA and the preliminary science examination (r=0.34, p<.001), the preliminary dental examination (r=0.27, p<.001) and the state examination (r=0.27, p<.001). Twelve percent of the performance variance for the preliminary science examination (NVP) were explained by the German GPA, and 7% for the preliminary dental examination (ZVP) as for the state examination. Studies performed in other European countries [8], [9], in Canada [10], [11], [12] or the USA [13], [14], [15] yielded similar results.

Subject-specific study aptitude tests measure cognitive abilities required for a specific subject of study [4], to make predictions of study success more reliable. The meta-analysis published by Hell et al. [3] reported a moderate, corrected validity of ρ=0.35 for the Test of Medical School (TMS) [16] from eight cumulative primary studies with a total population of 5.871 dental students, which explained 12.4% of the variance of study grades in preclinical and clinical dentistry. Embedding the TMS results into the German GPA significantly increased the accuracy of the study success prediction, even though GPA was the stronger predictor [5]. The dental study aptitude test “Besonderer Auswahltest Zahnmedizin” (BATZ, [17]) was performed in Germany during the 1970s and supports these findings. In 1978 Hitpass [17] found only low correlations between the ZVP examinations at the end of the fifth semester and GPA (r=0.25), respectively between ZVP and BATZ (r=0.34). However, if the two criteria are combined, there is a moderate correlation of r=0.42; the increase in the explained variance obtained with the BATZ was 11%.

Prior to the winter semester (WiSe) 2008/09, GPA was the only admission criterion for dental applicants at the University Medical Center Hamburg-Eppendorf (UKE). In order to improve the accuracy of study success predictions obtained from use of GPA, a subject-specific aptitude test entitled “The Hamburg Assessment Test for Medicine - Natural Sciences Test” (HAM-Nat) was developed for admission of medical and dental students [18].

The HAM-Nat is a multiple-choice test at high school level with a focus on aspects of mathematics, physics, chemistry and biology relevant for medical studies. Applicants receive a list of subjects to enable them to prepare for the HAM-Nat and, at the same time, for the syllabus of the first study semesters. Thus strongly motivated applicants were able to gain an advantage by intensive revision of their knowledge in these subjects [19].

The HAM-Nat was tested in 2006 and 2007 in the first study week [18] with medical and dental students, who were admitted by other criteria, before the paper based science test was first used for the selection of dental students in WiSe 2009/10. The 200 applicants with the best GPA, who had listed Hamburg as their first preference, were invited to take the HAM-Nat test. The admission criterion was a combination of HAM-Nat results (max. 59 points) and German GPA (max. 60 points). This meant that some applicants with a GPA as low as 2.2 obtained admission if they gained a good HAM-Nat result. In 2010 our selection procedure was extended by a wire-bending test (HAM-Man) to determine applicants’ manual abilities as a predictor of performance in the preclinical laboratory courses [20], [Manuscript in preparation].

There are already two validation studies for the HAM-Nat Natural Sciences Test in medicine. Hampe et al. [18] found a stronger prognostic validity for the HAM-Nat (R²=0.095, p<.001) as for the German GPA (R²=0.066, p<.001) with reference to preclinical academic success in the first two semesters. Inclusion of the HAM-Nat as an additional predictor over the German GPA yielded an incremental validity of 6.4%. In another HAM-Nat validation study Hissbach et al. [21] reported parallel-test reliabilities within the range of 0.53<r_t<0.67 and retest-reliabilities between 0.53<r_t<0.67. The authors also reported a correlation of r=-0.24 between HAM-Nat and GPA. The HAM-Nat test module “Scientific Thinking”, whose content and structure corresponds to the TMS module “Basic Medical-scientific Understanding”, correlated between r=0.21 and r=0.34 with various HAM-Nat versions.

The use of an additional selection instrument is of particular interest, as eight out of 29 German dental faculties were still using GPA as the only criterion for the selection of dental students in 2012. Only one other medical faculty, Witten-Herdecke, is using a self-developed aptitude test; seven others are using the TMS [22], [http://www.hochschulstart.de/index.php?id=3683]. It does not make much sense to use cognitive abilities as prediction a criterion for performance in preclinical laboratory courses [23] and that is why the HAM-Nat is not used in that case.

The present study investigates whether the prediction of the preclinical academic success is increased by the launch of the HAM-Nat in addition to GPA, so that its implementation in the selection of dental students is worth the effort, which means a better prognostic validity of our admission procedure. It is anticipated that the HAM-Nat will prove its worth as an additional selection instrument to GPA, which is why not only examinations in physics and chemistry, but also in NVP and the ZVP should be predicted by the HAM-Nat. The enhancement of the prognostic validity would justify the work and expense related to the HAM-Nat in the admission of dental students. It is impossible to address the question, whether the dropout of dental students can be reduced by the HAM-Nat as has been shows for the medical course [18], because the sample size is too small and the observation period is too short.

In the present study HAM-Nat and GPA of 2006, 2007, 2009 and 2010 are predictors of study success. The dependent variable “academic study success” is operationalized as grade points in physics and chemistry examinations as well as grades in NVP and ZVP examinations. The study population is subdivided into four cohorts (2006-2010). All participants gave written informed consent.

In 2006 the HAM-Nat test consisted of 52 multiple-choice items, which were increased to 60 in 2007. The dental freshmen admitted either by GPA or by waiting period quota were allowed 1.5 minutes for each question. Participation in the study was voluntary; there had been no preparation for the tests. The HAM-Nat was launched as a selection instrument in the dental admission procedure in 2009. The amount of multiple-choice questions was raised to 80. All applicants in 2009 and 2010 had free access to a list of HAM-Nat subject areas and two practice versions of the HAM-Nat via the UKE homepage.

The work and effort of the HAM-Nat is similar to a regular examination. Two faculty employees and four student assistants were in charge of the procedure. The lecture hall with a capacity for 200 applicants was provided free of charge by the University Hamburg. The questions were drafted in consultation with high school and university teaching staff. Answer sheets were evaluated by two faculty employees using high-resolution point scanners.

The dental freshmen (2006 and 2007 cohort) provided their GPA (1=very good to 4=satisfactory) in a separate questionnaire. In 2009 and 2010 the applicants’ GPA were provided by the “Stiftung für Hochschulzulassung” (SfH, Institute for University Admissions) in context of the admission procedure.

There are no recently published studies which show any prediction of course performance in the first semesters or exam subjects of the NVP and the ZVP separately; instead, most predictions are based on cumulative overall data [7], [8], [9], [10], [12], [13], [15]. This method leads only to average data on study performance that does not clearly identify individual prognostic effects for single examinations. Consequently, study success was operationalized as grade points for the examinations in physics in the first and in chemistry in the second semester. The NVP taken at the end of the second semester includes oral tests in physics, chemistry, and biology. The ZVP at the end of the fifth semester includes oral tests in anatomy, physiology, biochemistry, and dental prosthetics (ZEK). The ZEK examination is divided into an oral and a practical section, which are equally represented in the examination grade. The grades of the NVP and ZVP examinations are awarded on the same basis as in the German school system, that goes from 1 (very good) to 6 (insufficient).

The first step in the evaluation involves presentation of the descriptive data, including the Pearson correlations between GPA, the HAM-Nat and academic study success. Regression analysis is used to determine the influence of GPA and HAM-Nat as predictors of preclinical academic study success. The first regression model only contains GPA, but in the second regression model HAM-Nat data are included additionally to GPA. This proceeding allows the computation of predictive power for both predictors at the same time and hence the calculation of incremental validity of HAM-Man over GPA. The coefficient of determination R² indicates the percentage of explained variance in study performance by GPA and HAM-Nat. The standardized regression coefficients β can be used to compare the prognostic power of each predictor.

In order to identify biases in data analysis, the multiple regression models are tested for multicollinearity and auto-correlation of residuals. Durbin-Watson values approaching either to 0 or 4 indicate auto-correlation of the residuals. In order to preclude the presence of multi-collinearity [24], the variance inflation factor (VIF) must not exceed 10 and the tolerance levels must not lie below 0.2. The software PASW Statistics 18.03 was used for the statistical computations.

The demographic analysis just showed minimal differences in age and gender between the candidates of the four cohorts (see Table 1 [Tab. 1]). Only in cohort WiSe 2010/11 the gender ratio was roughly equal with 25 women and 23 men. In summary, the descriptive analysis revealed homogenous characteristics of the study variables across all four cohorts. The skewness, excess and significance of the Shapiro-Wilk statistics (α<0.05) indicated a non-parametric distribution for most of the variables (see Table 1 [Tab. 1]), but it has been shown, that a violation of normal distribution has very low influence on parameter estimation for correlation and regression coefficients. However, the rejection of normal distribution affects adversely the quality of the significance tests, which therefore should be interpreted cautiously [25], [26], [27].

The correlations between GPA and HAM-Nat were extremely low in all four cohorts (see Table 2 [Tab. 2]), indicating a sound discriminant validity. The linear relations of GPA and HAM-Nat as predictors of study performance differed between the four cohorts, and in some cases they even exhibited an inverse direction (see Table 2 [Tab. 2]). For example, the positive relation between GPA and physics examination grades in cohort 2009/10 (r=0.21, n.s.) indicated that a higher grade in the physics course corresponds to a lower GPA. Another example is the positive correlation between HAM-Nat and the NVP biology exam grade in cohort WiSe 2006/07 (r=0.16, n.s.), instead of the expected negative correlation, as displayed in cohort WiSe 2010/11 (r=-0.38, p<.05).

The results of the regression analyses are shown in Tables 3 to 5. Even though multi-collinearity and auto-correlation of residuals were not present, the significance tests must be interpreted with caution. As already observed in the correlation analysis, a few of the regression coefficients of GPA and HAM-Nat were negative, i.e. they indicated a direction for the relation between predictor and study performance that contradicts not only our expectations, but also the results obtained from other cohorts of our study.

Grade point average mostly explained less than 5% of the academic performance variance of subjects in the first two semesters (see Table 3 [Tab. 3]) and the NVP (see Table 4 [Tab. 4]), but there were also some prominent coefficients of determinations, e.g. for the chemistry course of the cohort WiSe 2006/07 (R²=0.204). The findings revealed a different characteristic for the HAM-Nat. Mostly more than 5% of the academic study performance variance in the first two semesters was explained by the science test (see Table 3 [Tab. 3]). Comparisons of standardized regression coefficients β showed that HAM-Nat proved to be a stronger predictor of academic performance in physics and chemistry in three of four cohorts than GPA. Similar findings were obtained for the performance predictions of the NVP exam subjects physics and chemistry, but those could not be confirmed in the prediction of the biology examinations (see Table 4 [Tab. 4]).

Only the predictions of academic performance in the ZVP examinations revealed different tendencies for both predictors within the three study cohorts (see Table 5 [Tab. 5]). In the cohorts of WiSe 2006/07 and 2007/08 (pilot testing) the difference of predicted academic performance was up to a maximum of 4% between GPA and HAM-Nat in the exam subjects anatomy, physiology and biochemistry, but in the cohort WiSe 2009/10 the percentage of predicted academic performance was 10% higher in anatomy and biochemistry and 6.4% higher in physiology for the HAM-Nat compared to the German GPA.

The present study investigated whether the selection of dental students was improved by the study aptitude test HAM-Nat. The findings did not prove to be consistent across all cohorts and preclinical examinations, which might be caused by the diverging characteristics of the cohorts. However, the findings revealed a clear trend indicating the HAM-Nat as an additional selection criterion to the German GPA in the admission of dental students, which improves the predictions of preclinical academic study success in dentistry.

All in all, the predictive power of HAM-Nat seems to be stronger as compared to GPA, especially for the study performance in physics and chemistry (see Table 3 [Tab. 3]). This could be due to the relatively low science proportion in the German GPA. The very low negative correlations between HAM-Nat and GPA (see Table 2 [Tab. 2]) showed high discriminant validity for both predictors. Therefore, HAM-Nat and GPA reflect different cognitive abilities and skills, which supports the use of both predictors in dental admission.

The results of the ZVP examination were significantly better predicted by the HAM-Nat in comparison to GPA, but just in cohort WiSe 2009/10 (see Table 5 [Tab. 5]). A possible explanation could be that the HAM-Nat test was already an essential element of the dental selection procedure in 2009 and participation was compulsory. It should therefore be presumed that students of this cohort prepared themselves intensively for the science test. Moreover, they had already been admitted partly on the basis of their science test results. However, GPA und HAM-Nat showed no differences in predictions of the ZEK examination in all three cohorts. This is probably attributable to the technical-practical section of the examination, in which students have to complete four practical examinations on a full-faced manikin with torso. Neither HAM-Nat nor GPA appears suitable for the prediction of performance in this type of examination.

The signs of single correlation and regression coefficients ran contrary not only to the aforementioned expectations, but also to results in other cohorts. The most probable explanation is the highly selective sample caused by the admission procedure and dropout in the first semesters (see Table 1 [Tab. 1]). The study variables have a rather asymmetrical range of values, which is reflected in single cases by inverse signs and extremely low values of the coefficients.

The data obtained in the present study supported the results for medical students of Hampe et al. [18], who reported predictions of preclinical study success in the single-digit percentage area by HAM-Nat and GPA. The prediction of preclinical study performance by HAM-Nat is similar to that obtained by TMS [16], but this finding is only of further interest in relation to the different test efficiencies of both study aptitude tests. The HAM-Nat test has an average duration of two hours, the participants need roughly six hours for completing the TMS.

It is evident from the literature and the present study that prediction of preclinical academic success in dentistry will only be possible to a low level with instruments like GPA and study aptitude tests. This could probably be explained by the fact that these selection criteria only reflect an applicant’s knowledge, skills and abilities at high school level. Consequently, the predictability of results obtained by dental students in examinations at university level appears rather limited compared to students’ motivation and their exam preparations, which should be more essential for predicting university examinations. Overall, the analyses of single examinations revealed a clearly differentiated picture for the prediction of preclinical study success compared to grade average of several examinations.

The application of the HAM-Nat in dental student selection substantially improves the prediction of preclinical academic success. The HAM-Nat can be utilized with low financial and organizational effort to improve selection decisions as well as offering dental study applicants with a GPA of 2.0 or above a realistic chance of admission.

The results of the present study are based on relatively small sample sizes. Therefore, variability of measurement data had a limited range caused by the invitation to the student selection procedure or the admission to study dentistry. Although the samples are representative, their characteristic did not fit all requirements for statistical methods.

We wish to thank the Dean of the Hamburg Medical Faculty, Prof. U. Koch-Gromus, for his support of our work group. The assistance provided by Mr. Alexander Vogelsang in compilation of datasets was indispensable. The datasets could not have been created without the support of Mr. Marco Böthern and the teaching staff of Hamburg University’s Chemistry and Physics Departments. We thank Ms. Julia Weinberg and Ms. Magdalena Sieversen for their patience in preparing the aggregate tables of results. This study was supported financially from the teaching promotion fund of the Dean’s Office of the Hamburg Medical Faculty and by the Federal German Ministry of Education and Research.

The authors declare that they have no competing interests.