Survey of operational performance capability in the course of development between the first and second threshold of vocational training

Objectives

Originally, the aim was to clarify the question of the “training maturity” (Ausbildungsreife) of young graduates of general education schools: can the category of “training maturity” be operationalised in an empirical research design in such a way that it can be answered coherently in terms of educational and developmental theory? The new aspect was that the methods used so far, such as TIMSS or PISA, ultimately take stock of the learning outcomes of the phases of preceding learning within the same milieu of knowledge. We wanted to try to capture the measurement construct referred to as operational performance capability from the perspective of the requirements to come.

In the project, which has been running since September 2008, we have meanwhile extended the research question to include the measurement of the effects of the vocationalisation of learning (EOL II and EOL III). In doing so, we respond to the first results, which suggest only a low level of maturity, whereas performance capability demonstrably increases in the milieu of vocational learning. We wanted to prove this effect by further developing the instrument presented with EOL I.

Research focus/research questions

The design of EOL II and III is to trace the vocationalisation of learning as a building-up of concepts that emerges from the subject matter but transcends its curricular steering and enrichment. We are particularly interested in the persistence of misconceptions and the conditions for their resolution. The ambitious result of the study should be to decide the question of whether it is the in-company or the school-based part of training that eliminates known misconceptions and is responsible for the superior performance of the vocational education system as a whole.

Methods

The questions are developed as plots and the answers as items for a multiple-choice procedure. In contrast to the (Rasch) scales of the usual studies and their ratings of “right” or “wrong”, we are testing the application of three rating categories: “right”, “everyday-theoretical” and “wrong”. In this way, we attempt methodologically to do justice to the ultimately guiding idea of a theory of education as a theory of development.

Results

A: Our experience suggests that the instruments used so far work because they deliver robust results. They involve increased effort, which has so far been justified by the fact that, in the evaluation tasks used on a trial basis, we obtained clear indications of the existence of so-called super-concepts, which make it possible to identify concepts effective below the level of the specific occupation that are built up in various domains but only developed fully in individual ones. This probably yields indicators that make vocational competence development comprehensible in such a way that the cross-domain significance of a concept becomes visible behind its specific use. This would mean, for example, that these concepts can only be developed within one vocational specialisation, but then acquire a more general, indeed an operational, significance. For the technical occupations, this has so far been undramatic. One could speak of a harmonious relationship between the results and the hypotheses.

This appears to be quite different in the commercial occupations. Apart from the very few exceptions, which were based on a workable arrangement constructed from the perspective of those involved, we were presented with solutions that for the most part were not based on any concept at all, in that the moral problem of a fair exchange, which first had to be identified by means of economic thinking, was not recognised, yet the solution was nonetheless heavily moralised.

B: We encountered an astonishing heterogeneity. Patrick Griffin, who heads the Assessment Research Centre (ARC) at the University of Melbourne and maintains close scientific contacts with international PISA experts and the OECD, has described similar observations, which probably also trouble the psychometricians. They achieve the comparatively high degree of homogeneity of their measurement findings by drawing on content on whose mastery the test subjects are grouped precisely into those groups that correspond to their level of performance. In a project commissioned by the American companies Cisco, Microsoft and Intel, he was to identify probable, future-relevant requirements and develop test instruments for assessments geared to them. What nobody expected was that the heterogeneity of the findings increased drastically – although his test candidates will belong to the pool of the coming elites of Cisco, Microsoft and Intel and thus can hardly have been among the weaker graduates of the relevant American universities, the differences in the measured performances were so large that, in view of the performance corridors psychometricians are used to, one began to doubt whether the right thing had been measured with suitable instruments.

Griffin’s interpretation of this problem concedes, on the one hand, that it is the simple effect that must occur when one gets closer to the educated person him- or herself than to his or her educational biography. In other words, for him the heterogeneity of the test results expresses the product of a development that is not random, but also not standardised or standardisable. In this respect, an important interim result can be noted: as soon as one leaves the past phase of individual development in order to obtain plots or items for performance measurement, the collective character of these completed phases is lost, and with it the homogenising effect that apparently emanates from it.

Further information

Bremer, Rainer 2010: Bericht zur EOL-Untersuchung zur Etablierung der OBF in Berlin. [Report on the EOL study on the establishment of the OBF in Berlin.]

id. 2012: Technik und Bildung – Versuch einer berufspädagogischen Antwort auf das erziehungswissenschaftliche Rationalitätsproblem [Technology and Education – an attempt at a vocational-pedagogical answer to the problem of rationality in educational science], Wetzlar (forthcoming).