gms | German Medical Science

In 2006, the Bachelor of Science in Nursing at the Bern University of Applied Sciences (BFH) was redesigned in accordance with the requirements of the Bologna Process. The degree programme is a competence-based general studies course, combining a scientific basis and consistent practical focus. The structure is modular; the pedagogic-didactic concept is Problem-based Learning (PBL). The teaching staff consists of twenty lecturers who have generally completed basic training in nursing and hold an academic degree in Nursing Science, Educational Science or Psychology. On average, 100 students are educated in Nursing Science every academic year.

Based on the evaluation literature which has proven itself well for PBL curriculums, the evaluation process of the degree programme was developed in parallel to the curriculum [1], [2]. The objective of the teaching evaluation (evaluations of modules, lecturers, etc.) was the continuous optimisation of the degree programme. This led to a corresponding evaluation concept, which can be characterised as follows:

1.

Continuous evaluation of all modules by students and lecturers for structural-organisational adaptations within the single modules.

2.

Continuous curriculum adaptation by the study group. This is the evaluation group, consisting of the head of the degree programme, the head of the training programme and the research associate. It sees to structural, content-orientated adaptations within the whole curriculum.

3.

Continuous quality development by the quality circle. The quality circle is composed of the entire teaching staff. It sees to the systemic-attitudinal adaptations, for example, concerning the basic educational attitude, for the realisation of the future professional profile.

At the end of every module, the students evaluated the module using a standardised questionnaire. The questions were designed to monitor the module quality. The students completed the questionnaire during a teaching session at the end of every module. This resulted in the very high return rate of 95 per cent. All evaluations were carried out anonymously.

The collected data was processed statistically and summarised. In the following module, the head of the degree programme presented the summary to the students and the results were discussed. The students received feedback as to whether and how their proposed changes or suggestions would be put into practice.

This approach has proven itself well in the past years. However, the question of whether the student survey should still be carried out anonymously led to controversial discussions among the staff. The reason for this was the appearance of single, noticeably negative numerical assessments, for example, rating all the questions with 1¹ or giving offensive and destructive comments such as “Mrs Müller² is completely incompetent! Send her to a further education course!” In addition, the very negative comments from the students were often not consistent with the corresponding numerical assessments. In such cases, it was not clear to the teaching staff how the results should be interpreted. Since the students evaluated the modules without giving their names, the lecturers could not check back with them, which led to great irritation. Part of the teaching staff suspected that anonymous evaluation favoured such very negative student responses. However, the amount of data available at the time was very small and therefore no well-founded conclusions could be drawn.

Supporters of the anonymous approach argued that students might fear that negative criticism could result in unpleasant consequences from lecturers. The research literature also points out that, particularly with sensitive or threatening answers that could have negative effects, there is a risk that the questionnaire will not be completed honestly but rather in accordance with social desirability [3]. Moreover, various studies have provided evidence that respondents tend to give more obliging answers if they know or suppose that the lecturers will be able to see their names alongside their answers [4], [5]. According to another study, carried out by Fries and McNinch [6], students do not give their names, even when asked to do so, whenever they have something negative to say. Therefore the specialised literature recommends that questionnaires should be completed anonymously in order to ensure the accuracy and reliability of the data [3].

However, there have been just as many studies that could not prove that there are any differences between anonymous and personalised data [7], [8]. Furthermore, the teaching staff adopts an attitude, in accordance with the basic educational concept, that students should be treated as adult partners and that an objective and open exchange regarding the performance quality must be possible. In the course of their studies, the students should learn to be able to give critical yet respectful and responsible feedback.

Previous research results on the subject of anonymous and personalised evaluation are inconsistent and some of the specialised literature is more than ten years old. This means that the question has not received much attention in recent years. The common approach to carrying out surveys anonymously – particularly when using psychometric methods – was therefore applied indiscriminately to programme evaluations. To be able to base the evaluation process on evidence and to close the current research gap regarding anonymous and personalised evaluation, those responsible decided to carry out a study. Its aim was to discuss the following question: Are there any differences between the outcomes of anonymous and personalised evaluations?

To answer this core research question, three sub-questions were defined, allowing a differentiated analysis regarding the research objective:

1.

Do the modules receive better quantitative assessment if the evaluations are anonymous or if they are personalised?

2.

Are more optional comments given in anonymous or personalised evaluation?

3.

Is the quality of the optional comments different in anonymous and personalised evaluation (see Chapter 2.3 “Data Processing”)?

The module evaluation questionnaires, specially designed for the curriculum, contain six items for the aspect Overall Impression of the module (objectives, relevance, specialism, subject matter, organisation, quality), four items for the aspect Lectures (didactic questions, lecturers, structure, quality, see Figure 1 [Fig. 1]) and four items for the aspect Exams (level, scale, content, quality). The students assessed all of these items on a scale from 1 (completely disagree) to 6 (completely agree). This scaling was chosen because it is consistent with the Swiss grading system and therefore demands little cognitive effort from the students while evaluating and because it allowed a tendency towards sufficient/insufficient to be observed. In addition, depending on the structure of the individual modules, different items were questioned, e.g. concerning tutorials, skill trainings, practical courses or seminars, see Figure 1 [Fig. 1] [1], [2], [9], [http://www.qualitaet.unibe.ch/unibe/qualitaet/content/e4198/e4436/e4456/MusterfragebogenTheologieVorlesung_ger.pdf].

To be able to subsequently analyse the research question, an additional six-level scale was designed for every aspect by averaging its items.

In addition to the module assessment using the given scale, the students had the possibility to make comments and/or suggestions about every aspect. This means that the students could make more than one optional comment in the questionnaire.

For the analysis of the research question the evaluation setting was changed: one half of the students still completed the questionnaires anonymously, whereas the other half completed questionnaires with their names printed on them. The data collection for this study took two years. The sample was composed of students from four cohorts. For every module evaluation the students were randomly re-assigned either to the “anonymous” group or the “personalised” group.

During the study period a total of 27 modules were evaluated. All the practice and communication modules as well as the Clinical Assessment modules were excluded from the study because other questionnaires were used to evaluate these. The people in charge of the study chose five out of the 27 evaluated modules that suitably represent the multifaceted Bachelor in Nursing: “The study of nursing (introductory module)”, “Dealing with emergency situations”, “Understanding research”, “Acquiring basic statistical knowledge” and “Ensuring nursing quality”. Altogether 615 completed questionnaires, of which 306 were completed anonymously, were used in the study.

The analyses of the answers were carried out individually and independently for each module. Since the groups for every module evaluation had been randomised anew, it was quite likely that the same student assessed one module anonymously and another module with a personalised questionnaire.

For the first two sub-questions, the data was entered into the statistical programme SPSS and the analysis carried out as described in Chapter 3 “Results”.

For the third sub-question the data had to be processed separately. In order to do this, all the comments were assessed using a rating instrument developed by one of the authors. The instrument was developed with the objective of coding statements in a way that shows whether they are offensive or destructive. First, the common features of offensive or destructive comments had to be considered. Based on this, it was defined that a statement is offensive or destructive only if it meets all of the following five conditions: it is

1.

negative,

2.

absolute,

3.

refers to a person,

4.

contains an order and

5.

is visually accentuated.

The five resulting dimensions were defined as follows:

1.

In the dimension Assessment the statements were coded as positive, negative or both, e.g. “The module was poorly organised, but the topics were all very interesting”.

2.

In the dimension Differentiation it was assessed whether the statements were differentiated or absolute. A statement, for example, was considered differentiated if it contained words like “partly”, “sometimes”, “but”, etc. Statements like: “The topic of normal distribution is unimportant” were coded as absolute.

3.

The dimension Individual showed whether a statement referred to a person, namely a lecturer, and whether her/his name was mentioned, e.g. “Mrs Müller is incompetent!”

4.

In the dimension Order it was assessed whether the responses contained an order, e.g. “Explain to the associate lecturers once and for all how the digital projector works”.

5.

In the dimension Visual Accentuation, statements were coded as to whether they were accentuated with punctuation like an exclamation mark or highlighting, e.g. “Super!!!” or “HELLO PLANNING!”.

Using the rating instrument presented above, three people independently coded all comments and suggestions into categorical variables and analysed and assessed them. During the coding process the assessors did not know whether a statement came from the anonymous or the personalised group. To verify the assessors’ consistency, the Pearson correlation coefficient was calculated. The value of the consistency for the single modules was between r=.74 and r=.98.

To analyse whether anonymous evaluations tended to be more negative than personalised evaluations, a multivariate variance analysis (MANOVA) was carried out for every single module. The scales developed according to the described aspects of the module evaluation were the dependent variables; the grouping variable “anonymous” and “personalised” was the independent variable.

For none of the modules could any significant difference between anonymous and personalised data be identified (see Table 1 [Tab. 1]).

After a simple frequency count of the optional comments, a Mann-Whitney test for independent samples was carried out for every module to verify whether the students evaluating anonymously tended to make optional comments more frequently. The Mann-Whitney test was chosen because the data did not meet the requirements for a t-test.

As can be seen in Table 2 [Tab. 2], the students in the personalised group tended to make suggestions and comments more often. However, the frequency does not differ significantly from one group to the other. The optional comments in the module “Understanding research” can be considered as an exception: in this module, the personalised group made suggestions or expressed opinions significantly more often on average.

The students formulated a total of n=2152 statements about the five modules relevant for the study; n=6 of these were offensive or destructive, meaning they met all of the above-mentioned conditions (see Chapter 2.3 “Data Processing”). Due to this small number of statements meeting the conditions of an offensive or destructive statement, it was not possible to carry out a statistical analysis to determine whether the students evaluating anonymously made offensive or destructive statements more often.

As a first step for the statistical analysis, the cumulative value per dimension was established for all the optional comments in the questionnaire. This means that if a student made three non-personal comments in one evaluation, he or she received a cumulative value of 3 on the dimension “Individual”. Thus the lower the cumulative value, the more often the students wrote responses that were negative, absolute or personal or that contained an order or a visual accentuation.

As a second step, the data was checked for possible differences between the groups in the quality of the optional comments using MANOVA (IV=grouping variable anonymous vs. personalised, DVs=cumulative values of all dimensions) (see Table 3 [Tab. 3]).

No significant qualitative differences between anonymous and personalised comments could be identified in any of the five dimensions.

The objective of the study was to determine whether there is a difference between anonymous and personalised programme evaluation by students. As a result it could be shown that, regarding the quantitative assessments, there are no differences between the two analysed groups. Due to the small amount of data it could not be determined whether there is a difference concerning offensive or destructive statements, but an analysis of the characteristics of the five qualitative dimensions shows no significant difference between anonymous and personalised evaluation. The frequency of the optional comments differed significantly only for one module; for all the other modules no difference between the anonymous and personalised group could be found. These results leave much room for interpretation. The single significant result might be explained by the fact that students who make an effort to formulate an elaborate personal comment would like to be perceived as individuals. They would like to initiate a further exchange.

The consistent feedback cycle presented in Chapter 1.1 “Initial Situation” and the high degree of transparency in the handling of the results gives the students the certainty that their feedback will be taken seriously and considered carefully. The students therefore do not abuse questionnaires to vent frustration about other issues.

In addition to the standardised module evaluation other feedback possibilities exist, notably a mentoring concept that facilitates individual exchange between students and lecturers. Possible dissatisfactions or problems can thus be addressed specifically and in various ways. Another explanation for the present results might be that the lecturers signalised that feedback and suggestions for the development of the programme would be welcome.

This evaluation concept affects lecturers as well as students. Both still see themselves, to a certain extent, as pioneers of the new degree programme and would like to contribute to the curriculum design. Furthermore it can be assumed that rules for communication and feedback are fully integrated in the curricula of the feeder schools, where the higher education entrance qualification is obtained. The generation studying today is used to communication and discussion.

Based on the results of the study, offensive and destructive statements must be viewed as isolated cases. The lecturers were unaware of this fact when the evaluation concept was first used (see Chapter 1.1 “Initial Situation”), and the amount of data available at the time did not allow any universally applicable conclusions to be drawn. Outliers were not recognised as such and therefore overrated. It was suspected that they came about because of the use of anonymity.

The theory referred to as “Negativity Bias” in the literature confirms that in most situations something negative will be perceived more strongly, dominantly and drastically than something positive [10]. Royman and Rozin [11] illustrate this theory with the comparison that the slightest contact with a cockroach makes a delicious meal inedible. According to the principle of “Negativity Dominance” [11][, the perception and judgement of events that feature both positive and negative aspects are more negative than the arithmetical sum of these subjective values.

Numerous other reasons may be accountable for single negative statements: professional overload, private problems, lack of interest in a subject or a “Negativity Bias” on the students’ side, for example, one exam question that is too difficult makes the whole module bad.

As a limitation of the present study one could name the self-elaborated design of the five-dimensional instrument to determine the statement quality of the optional comments, since this categorisation is subjective and cannot be exhaustive. Nevertheless, the high assessment consistency between the independent assessors is indicative of the validity of the instrument. It should also be noted that it is precisely the use of this rating instrument and the assessment of the optional comments that made it possible to gain an insight into the students’ critical engagement with the curriculum.

A strength of the study lies in the large amount of data available: due to the comprehensive implementation of the module evaluation, the response rate was almost 100 per cent. This in turn was only made possible by the well-thought-out concept. Over the years, the evaluation questions have proven to be relevant. All the careful preliminary work and the well-practised setting made a smooth implementation of the research project possible.

It is worthwhile to invest in the development of an evaluation concept whose central aspect is a closed and transparent feedback cycle. This provides the students with the possibility of participation and shows them how the evaluated data is integrated into the continuous development of the course offer. What is crucial is not the number of evaluations – although a minimum should certainly be ensured – but rather the consistent reporting of the results to the students and the discussing of possible measures. The study shows that it does not matter whether an evaluation is carried out anonymously or in a personalised way, as long as these conceptual evaluation conditions are met. With this in mind, the results of the present study can be applied to other academic institutions.

Extremely negative responses must be put into perspective, since negativity seems to have a stronger effect than positivity. This is a further, possibly relieving, conclusion.

¹ Swiss grading system: 6 = excellent, 5 = good, 4 = pass, 3 = fail, 2 = very poor, 1 = no performance

² Real name withheld.

The authors declare that they have no competing interests.