Questions about the validity of the UMAT

A letter in the most recent MJA inside the box below

To the Editor: By focusing on the observed low correlations between the Undergraduate Medicine and Health Sciences Admission Test (UMAT) and academic scores at the University of Queensland (UQ), Wilkinson and colleagues1 conclude that the UMAT is a poor predictor of medical student performance. Alternatively, one could focus on the very high mean and low variance grade point average (GPA) scores of the UQ student cohort and conclude that the use of UMAT scores was clearly very successful in identifying a high-performing group of students.

However, before making any conclusions based on these data, we should draw on the vast literature that highlights common problems that significantly affect validity coefficients.2 Range restriction is one such problem, and Wilkinson and colleagues rightly used Thorndyke case 2 to correct for this in their UMAT scores. Although they achieved little change in observed correlation values, a recalculation using the mean standard deviation of the entire UMAT cohort for the period 2005–2009 actually takes the 0.14 correlation between Section 1 and Year 1 GPA to 0.27. Despite this considerable increase, Thorndyke case 2 only accounts for direct range restriction of the predictor (UMAT scores). However, in the context of medical student selection, there are other important sources of error that need to be accounted for in order to better appreciate predictive validity.

First, use of a high cut-off on the high school examination score is very likely to add indirect range restriction on the UQ UMAT scores. This may be further attenuated by not including measures of non-cognitive ability (interviews). Second, selection on cognitive ability tests (UMAT and high school results) also creates range restriction on the largely cognitive criterion (GPA) that would further reduce observed validity correlations. The UQ study illustrates such restriction. Third, the unreliability of the criterion itself needs to be accounted for, especially when more subjective ratings are included, such as ratings of clinical performance.

The problem of relatively small samples in selection studies is well documented.2 Given the evidence of extreme range restriction of predictor and criterion variables used in medical school selection research, more primary studies (followed by meta-analysis) are necessary before accurate conclusions can be made about the use of the UMAT, or of any other predictor used in this high-stakes selection context.

Basically talking about the range restriction limitation in most research done on the UMAT: that is can you actually meaningfully assess the usefulness of the UMAT when you're researching a bunch of students that have a high UMAT and high GPA/ATAR 'cause maybe the UMAT is useful at selecting out good medical students but its usefulness reaches a threshold by the time you get to the top 10% or whatever where increasing scores don't produce increasing usefulness. A fair a logical criticism, I think.

Arguably, UMAT isn't designed to be specific for academic high achievers which is what the study was looking at - if that were truly the case then UMAT would be scrapped and they'd simply use high school marks instead.

Originally Posted by Havox

Arguably, UMAT isn't designed to be specific for academic high achievers which is what the study was looking at - if that were truly the case then UMAT would be scrapped and they'd simply use high school marks instead.

I don't think that's entirely true; the UMAT wasn't designed solely for very high academic achievers, but it was designed with them in mind. Arguably, the applicant pool the UMAT should have been/is designed for is that of the upper-ish quartile (i.e. 90+ ATAR) and is meant to demonstrate abilities (or a lack of) for applicants in that range. On another note, I don't think it necessarily needs changing to allow lower achieving students in if it's still doing it's job.

Originally Posted by Bobby21

But the point is, there is a chance of interviewer bias/prejudice influencing a candidate's score and that is unacceptable. I think interviews with multiple stations are the best compromise in that they remove much of this chance while still retaining their purpose.

Lets face it, every single human being on this planet WILL have SOME degree of bias as a result of their collective experiences. Lets not forget that these interviewers have interviewed (probably) thousands of candidates, based on that alone I'd say that their sheer amount of experience in weeding out the pretentious douche bags from the genuinely good doctors will make up for their bias'.

To gain more clarity on the matter, could you provide an example of something you would consider to be bias? The reason I ask this is because I'm assuming you're not referring to just a general dislike towards a certain race of people (I doubt an interviewer would be so shallow) but that you have some deep-seated view on this issue.

Originally Posted by TheMedHopeful

Lets face it, every single human being on this planet WILL have SOME degree of bias as a result of their collective experiences. Lets not forget that these interviewers have interviewed (probably) thousands of candidates, based on that alone I'd say that their sheer amount of experience in weeding out the pretentious douche bags from the genuinely good doctors will make up for their bias'.

To gain more clarity on the matter, could you provide an example of something you would consider to be bias? The reason I ask this is because I'm assuming you're not referring to just a general dislike towards a certain race of people (I doubt an interviewer would be so shallow) but that you have some deep-seated view on this issue.

Actually, many interviewers do not have experience interviewing candidates for medical school. They do usually undergo some form of training, however. While everyone has bias, the point bobby21 was making was that this bias can be diluted through the use of multiple interview stations. I'll answer for bobby21 on the issue of bias examples would include bias against those from private schools, those of Asian background, bias in favour of candidates from their own ethnic background (who they find easier to relate to), bias against those with religious beliefs, bias against those with strong opinions.

Bias is not necessarily bad either, for example, should and interviewer be biased against someone who is well-mannered but who does not necessarily converse in a discourse of familiarity and casual confidence?

As helpful as your comment was, please do not post padd on MSO. Thanks