Critique of a Study on “Racism in Recruitment”

Post Views: 787

Critique of Study on “Racism in Recruitment”

An organisation called Centre for Governance and Political Studies (https://centgps.com.my/) recently released a report detailing their study on racial biases found in job call-backs in Malaysia.

Although it seemed promising at first, my friend Sabrena Arosh quickly pointed out to me the gaping flaws in their research based on an online article. Once I got my hands on the full report, I too, became extremely disappointed in their poor research methodology.

Furthermore, as proud HELP Psychology graduates, my friend and I felt quite strongly about this because we are researchers ourselves. And it is extremely frustrating to see bad research practice misinforming the public.

The following is an extensive critique aimed primarily at the research and the methodology behind a recent local study entitled: “Racism in Recruitment: A Study on Racial Bias for Entry Level Jobs in Malaysia”.

You can find the full report here: https://centgps.com.my/wp-content/uploads/2019/03/Racism-in-Recruitment-A-Study-on-Racial-Bias-For-Entry-Level-Jobs-in-Malaysia-.pdf

CRITIQUE PART 1: INTRODUCTION

1. There is a distinct lack of citations and references within the entirety of the document.

Why is this a problem?

The lack of citations and references is unscientific, as there is no traceable, basis or evidence for their claims.

2. In the ‘Goal’ section, the authors state: ‘our study revolved around the assumption…’

BUT!

Good research doesn’t emerge from assumptions. It emerges from comprehensive, exhaustive, and critical reviews of existing literature.

3. Using anecdotal reports to make confident claims.

Referring to page 9 of the report, the authors describe several instances of discrimination in the office. However, this is purely anecdotal as it is based on a single reporter’s interview. In addition to that, the source is not clearly identified or cited. Moreover, based on those few interviews, the researchers conclude that: “…evidence of it happening in Malaysia has been made clear.”

Why is this a problem?

Before, in my undergraduate research classes, I teach my students that good research is tentative because it has to be falsifiable. Specifically, having definitive claims such as this only serves to mislead the reader. Anecdotal evidence easily leads to confirmation bias and thus, it should never be used to make concrete claims.

CRITIQUE PART 2: PAST STUDIES

4. All results that are presented to support their claims are from survey studies without any form of significance testing.

What is significance testing?

Significance testing tells you if your results are due to a true cause-and-effect relationship or have randomly occurred due to chance. Furthermore, while survey studies have their time and place in research as well, but the authors seem to rely too much on survey data and percentages to back up their claims instead of more grounded research.

5. The study relies heavily on evidence found in a previous study by Dr. Hwok-Aun Lee, and Dr. Muhammad Abdul Kahlid.

This current study is not even parallel against that study as Dr Hwok-Aun and Dr. Muhammad tested their hypotheses with regression analyses to determine the statistical significance of the differences they found, thus giving his claims more scientific credibility. In addition, they both also considered the ethnicity of the main controllers of the companies to determine if it was also another factor at play which was not done in this current study.

Why is this an issue?

The current study claimed in their Goal (page 6) that they wanted to confirm if discrimination still happens in the workplace, based on the evidence of discrimination in Dr Hwok-Aun and Dr. Muhammad’s study. However, their own study does not effectively replicate the same level of control in their methodology or statistical standards in their data analysis when compared to the previous study.

You can read Dr Hwok-Aun and Dr. Muhammad’s study here: https://umexpert.um.edu.my/file/publication/00006071_99752.pdf

6. Referring to page 12, paragraph 6, it is mentioned that the racial composition of the companies being applied to is an important indicator that needs to be considered within the context of the current study.

BUT!

It is not at all taken into account (or there is no clear evidence of this in the report).

CRITIQUE PART 3: METHODOLOGY

7. Referring to page 15 where the authors present the creation of the resumes for submission, it is clearly stated that the use of pictures is a team decision.

And why is this an issue?

There is no scientific justification for the use of the pictures on the resume. Thus, a group of people agreeing on something does not make for informed decision making.

8. Referring to page 28, paragraph 3, it is unclear what specific university qualifications were afforded to each resume. It is only stated that the applicants had ‘almost similar’ academic scores, and they provided the name of a ‘recognised and respected private college’.

But why is this a problem?

Research methodology should be clear and precise so that we can replicate the studies, because accurate replication is a key part of scientific research. However, in the case of this current study, the specific details of the resumes are unclear. Moreover, considering that the resumes are a key part of this study, it’s odd that they did not provide more specific information (or even examples of the resumes).

CRITIQUE PART 4: OUTCOMES

9. Referring to page 30, paragraph 3, the authors claim that their sample size was guided by feeling: ‘Our team felt that the sample size was sufficient.’

The authors also claimed that the sample size ‘represent the total number of available jobs for fresh graduates from December 2018 to February 2019.’

So why is this an issue?

Good research bases a sample size on evidence from previous studies or sample size recommendations. However, the number of actual available jobs available from Dec 2018 to Feb 2019 was not provided and hence we cannot verify if their sample size was indeed representative or not.

10. Referring to page 39, paragraph 1, the authors describe how the inclusion of the hijab option was a ‘calculated decision’ by the research team.

Remember a similar previous issue?

Once again, no scientific evidence is provided on how a hijab might possibly lead to job discrimination. And researchers should provide reason for including variables in their experiments. While it might certainly be interesting to look into the effect of wearing a hijab on job discrimination, without some form of guiding theory or conceptual framework, interpretations of the results can run in almost any direction.

11. Referring to page 41, paragraph 2, the authors said that a Malay girl who doesn’t wear a hijab has a 60% advantage in getting job interviews.

BUT!

If you look at page 39, the table actually says that the advantage is only 40%. Why the reduction?

12. Under ‘Additional Input’ on page 42, there are other statements meant to colour the reader’s impression of the study without any explanation or reasoning for their inclusion.

Why is this an issue?

Statements like the one’s found on page 42, when given without context or supporting evidence, seem to suggest researcher bias. Because academic writing should not be coloured by bias. But rather, it should provide a balanced view.

13. Lack of comprehensive statistical analysis.

The current study relies solely on descriptive statistics (i.e, whole numbers, ratios, and percentages) to draw conclusions.

Why is this a problem?

Descriptive statistics, as the name implies, can only describe the data. We cannot use it to draw firm conclusions. And that’s a job for inferential statistics and hypothesis testing. Rather disappointingly, the current study completely lacks this.

CONCLUSIONS OF THIS CRITIQUE

To conclude, this paper shows a lack of application of good scientific research methods, and a substantial degree of sweeping statements peppered with emotional rhetoric as perfectly exampled by the conclusion on page 46.

Good research should feature an extensive and unbiased literature review, good and precise methodology, as well as appropriate data analysis. Unfortunately, the current study is lacking in all these areas in their attempt to shed light into discrimination in the workplace.