Medical & Dental School Admissions: Some More Research

Last year there was a fairly large discussion on this forum surrounding a study from UQ that found UMAT to be of no predictive value in the early years of a medical degree (AMA - Medical admission test is not a valid predictor of academic performance). One of the issues there was that the study had some major flaws/weaknesses, but still, MSO members seemed interested.

I've come across a few similar studies that have been published since which I reckon are slightly more robust (although there are still major issues with trying to evaluate admissions procedures in the way they've done so). Figuring that at least a few people here would be interested, the references and abstracts are below.

All of these studies were published within the last 12 months and aimed to investigate (in some way) the validity/effect of GPA/UMAT/interview as components of medical/dental school selection in Australia/NZ. If you know of any other studies that are similar and potentially relevant to MSO members, feel free to post as a reply to this thread.



Mercer A, Puddey IB 2011. Admission selection criteria as predictors of outcomes in an undergraduate medical course: A prospective study. Med Teach 33:997-1004

Background: In 1998, a new selection process which utilised an aptitude test and an interview in addition to previous academic achievement was introduced into an Australian undergraduate medical course.
Aims: To test the outcomes of the selection criteria over an 11-year period.
Methods: 1174 students who entered the course from secondary school and who enrolled in the MBBS from 1999 through 2009 were studied in relation to specific course outcomes. Regression analyses using entry scores, sex and age as independent variables were tested for their relative value in predicting subsequent academic performance in the 6-year course. The main outcome measures were assessed by weighted average mark for each academic year level; together with results in specific units, defined as either ‘knowledge’-based or ‘clinically’ based.
Results: Previous academic performance and female sex were the major independent positive predictors of performance in the course. The interview score showed positive predictive power during the latter years of the course and in a range of ‘clinically’ based units. This relationship was mediated predominantly by the score for communication skills.
Conclusions: Results support combining prior academic achievement with the assessment of communication skills in a structured interview as selection criteria into this undergraduate medical course.

Poole P, Shulruf B, Rudland J, Wilkinson T. 2012. Comparison of UMAT scores and GPA in prediction of performance in medical school: a national study. Med Educ 46:163-171

Context  Medical schools continue to seek robust ways to select students with the greatest aptitude for medical education, training and practice. Tests of general cognition are used in combination with markers of prior academic achievement and other tools, although their predictive validity is unknown. This study compared the predictive validity of the Undergraduate Medicine and Health Sciences Admission Test (UMAT), the admission grade point average (GPA), and a combination of both, on outcomes in all years of two medical programmes.
Methods  Subjects were students (n = 1346) selected since 2003 using UMAT scores and attending either of New Zealand’s two medical schools. Regression models incorporated demographic data, UMAT scores, admission GPA and performance on routine assessments.
Results  Despite the different weightings of UMAT used in selection at the two institutions and minor variations in student demographics and programmes, results across institutions were similar. The net predictive power of admission GPA was highest for outcomes in Years 2 and 5 of the 6-year programme, accounting for 17–35% of the variance; UMAT score accounted for < 10%. The highest predictive power of the UMAT score was 9.9% for a Year 5 written examination. Combining UMAT score with admission GPA improved predictive power slightly across all outcomes. Neither UMAT score nor admission GPA predicted outcomes in the final trainee intern year well, although grading bands for this year were broad and numbers smaller.
Conclusions  The ability of the general cognitive test UMAT to predict outcomes in major assessments within medical programmes is relatively minor in comparison with that of the admission GPA, but the UMAT score adds a small amount of predictive power when it is used in combination with the GPA. However, UMAT scores may predict outcomes not studied here, which underscores the need for further validation studies in a range of settings.

Rich, A. M., Ayers, K. M. S., Thomson, W. M., Sinclair, R. J., Rohan, M. J. and Seymour, G. J. 2012. Does performance in selection processes predict performance as a dental student?. European Journal of Dental Education, 16: 27–34.

Objective:  This study investigated associations between the performance of dental students in each of the three components of the selection procedure [academic average, Undergraduate Medicine and Health Sciences Admission Test (UMAT) and structured interview], socio-demographic characteristics and their academic success in an undergraduate dental surgery programme.
Materials and methods:  Longitudinal review of admissions data relating to students entering dental education at the University of Otago, New Zealand, between 2004 and 2009 was compared with academic performance throughout the dental programme.
Results and discussion:  After controlling for variables, pre-admission academic average, UMAT scores and interview performance did not predict performance as a dental student. Class place in second year, however, was a strong predictor of class place in final year. Multivariate analysis demonstrated that the best predictors of higher class placement in the final year were New Zealand European ethnicity and domestic (rather than international) student status. Other socio-demographic characteristics were not associated with performance. These interim findings provide a sound base for the ongoing study.
Conclusion:  The study found important socio-demographic differences in pre-admission test scores, but those scores did not predict performance in the dental programme, whether measured in second year or in final year.

Puddey IB, Mercer A, Carr SE, Louden W. 2011. Potential influence of selection criteria on the demographic composition of students in an Australian medical school. BMC Medical Education, 11:97

Background
Prior to 1999 students entering our MBBS course were selected on academic performance alone. We have now evaluated the impact on the demographics of subsequent cohorts of our standard entry students (those entering directly from high school) of the addition to the selection process of an aptitude test (UMAT), a highly structured interview and a rural incentive program.
Methods
Students entering from 1985 to 1998, selected on academic performance alone (N = 1402), were compared to those from 1999 to 2011, selected on the basis of a combination of academic performance, interview score, and UMAT score together with the progressive introduction of a rural special entry pathway (N = 1437).
Results
Males decreased from 57% to 45% of the cohort, students of NE or SE Asian origin decreased from 30% to 13%, students born in Oceania increased from 52% to 69%, students of rural origin from 5% to 21% and those from independent high schools from 56% to 66%. The proportion of students from high schools with relative socio-educational disadvantage remained unchanged at approximately 10%. The changes reflect in part increasing numbers of female and independent high school applicants and the increasing rural quota. However, they were also associated with higher interview scores in females vs males and lower interview scores in those of NE and SE Asian origin compared to those born in Oceania or the UK. Total UMAT scores were unrelated to gender or region of origin.
Conclusions
The revised selection processes had no impact on student representation from schools with relative socio-educational disadvantage. However, the introduction of special entry quotas for students of rural origin and a structured interview, but not an aptitude test, were associated with a change in gender balance and ethnicity of students in an Australian undergraduate MBBS course.

Shulruf B, Poole P, Ying Wang G, Rudland J, Wilkinson T. 2011. How well do selection tools predict performance later in a medical programme? Advances in Health Sciences Education, Advances in Health Sciences Education, Online First

The choice of tools with which to select medical students is complex and controversial. This study aimed to identify the extent to which scores on each of three admission tools (Admission GPA, UMAT and structured interview) predicted the outcomes of the first major clinical year (Y4) of a 6 year medical programme. Data from three student cohorts (n = 324) were analysed using regression analyses. The Admission GPA was the best predictor of academic achievement in years 2 and 3 with regression coefficients (B) of 1.31 and 0.9 respectively (each P < 0.001). Furthermore, Admission GPA predicted whether or not a student was likely to earn ‘Distinction’ rather than ‘Pass’ in year 4. In comparison, UMAT and interview showed low predictive ability for any outcomes. Interview scores correlated negatively with those on the other tools. None of the tools predicted failure to complete year 4 on time, but only 3% of students fell into this category. Prior academic achievement remains the best measure of subsequent student achievement within a medical programme. Interview scores have little predictive value. Future directions include longer term studies of what UMAT predicts, and of novel ways to combine selection tools to achieve the optimum student cohort.

Comments?

I'm a little surprised that interview scores have little predictive value. All down to human error perhaps. And the ability of some to put on a show better than others.

students of NE or SE Asian origin decreased from 30% to 13%, students born in Oceania increased from 52% to 69%

Thats all my speculation cleared up

Speculation? Fairly well known phenomenon, which has just been quantified officially, no?

Originally Posted by greenglacier

students of NE or SE Asian origin decreased from 30% to 13%, students born in Oceania increased from 52% to 69%

What is meant by this? Does origin refer to place of birth of the student or whether the students were immigrants versus born Australians.

Originally Posted by chinaski

Speculation? Fairly well known phenomenon, which has just been quantified officially, no?

Well I had not personally done any research into this and I did not dare use my (gazillion tonnes of) anecdotal data to jump to a conclusion. Good to see its not just me. But yes your completely right just been quantified officially.

@godoftoast -What is meant by this? Does origin refer to place of birth of the student or whether the students were immigrants versus born Australians.

By origin I assume ethnicity regardless of whether they are immigrants/citizens/born. Some discrimination here would be very valuable.

Originally Posted by bobby190

By origin I assume ethnicity regardless of whether they are immigrants/citizens/born. Some discrimination here would be very valuable.

To what end, out of interest?

Originally Posted by bobby190

By origin I assume ethnicity regardless of whether they are immigrants/citizens/born. Some discrimination here would be very valuable.

Do you perhaps mean distinction or delineation?

Well it is evident that NE/SE asian students are disadvantaged by the introduction of UMAT. It would be good to see why. A lot of my freinds complain about cultural differences in S2 of UMAT, im not sure if its just banter or real. It might be able to give insight as to whether this aptitude is genetically disadvantaging asian population, or whether what is learned (cultural) is limiting them.

ETA: @BRS same diff right? Think they are synonyms

I tend to see it a different way. If they are under-performing in one section of selections, why not look at why they are under-performing, rather than focussing on this so-called "disadvantage" (ie look at the candidate, not the exam)? This stems further than just UMAT sections; the ethnic demographics of graduate entry have changed significantly also. If there are cultural differences hindering students at this stage, is it not reasonable to expect it to continue to dog them throughout their early career?