A cognitive assessment/aptitude test may or may not be reliable depending upon the type and level of questions included in the assessment.
What do we mean by a 'reliable' Aptitude Test?
It simply means that the cognitive assessment/aptitude test is able to identify the right talent needed in a job role- the right talent quality and quantity.
- The talent quality should be optimum. Neither over intelligent or less intelligent than the job role requires.
- The talent quantity should be optimum too. Not too many. Not too less.
So how does a cognitive assessment/aptitude test meet all the above conditions?
The answer is- by ensuring that the questions included in the cognitive assessment meet certain criteria.
Criteria 1: Right type of questions
The First Step in the process of creating accurate aptitude tests is getting the Question Type right.
To identify if the questions included in the assessment are of the right type, it's important to know the kind of cognitive abilities a candidate needs to have to perform well in any job role.
You must be wondering- ‘What are cognitive abilities?’
Cognitive abilities are skills that make up our IQ or Mental Aptitude. These skills include numerical reasoning, verbal ability, data analysis, logical & abstract reasoning, critical thinking and problem-solving.
Each cognitive ability that a candidate requires to succeed in any job role is called a 'Cognitive Competency.'
The key to ensuring that the type of questions included in a cognitive assessment for a particular job role is right is basing them on the cognitive competency framework for that job role.
For example: For a 'Data Analyst' job role, the cognitive competencies that a candidate requires are: numerical reasoning and data interpretation. Hence these competencies make up the cognitive competency framework for the job role of 'Data Analyst.' The cognitive assessment that is being used to screen candidates for this job role should have questions on these two competencies.
Criteria 2: Right level of questions
To determine if the questions included in the cognitive assessment are not too easy or too difficult for the talent pool, we need to check three things:
- Assessment Mean Score
- Difficulty Index of each question
- Discrimination Index of each question
1)Assessment Mean Score
The Second Step to creating an accurate aptitude test involves getting the Assessment Mean Score right.
Mean Score is the average score that candidate pool giving the same cognitive assessment achieved. The mean score shouldn’t be too high or too low as that means that the assessment is either too easy or too difficult for the target audience. In both situations, the cognitive assessment will not be able to identify the right talent for the job role. If the test is of 100 marks, the mean score should be between 45-55.
- Low mean score signifies that less number of candidates from the talent pool were able to score high on the aptitude test.
- Medium mean score means that a moderate number of candidates were able to score high on the aptitude test
- High mean score means that the majority of candidates from the talent pool were able to score high on the aptitude test
The ideal mean score for you depends upon what level and the number of candidates you are looking for.
e.g., Out of your talent pool, If you want to select very few candidates with the highest cognitive intelligence, your cognitive assessment should have a very low mean score.
Note: You can adjust the mean score to adjust the number and IQ level of candidates. For example, if you want to select the top 5 percentile of the talent pool, you should aim for an aptitude test a with mean score of 5%. It is a tool used for benchmarking an aptitude test as per an organization’s requirement.
Disclaimer: The kind of talent you end up with depends on the quality of the talent pool, in general. If your talent pool quality in general is poor, selecting even the top 1 percentile may not provide you with high IQ candidates. Vise-a -versa, if your talent pool quality is too high, even lower percentile candidates may turn out to be high in IQ as the average IQ of the talent pool is high.
2)Difficulty Index of each question
The Third Step to creating an accurate aptitude test involves getting the Assessment Difficulty Index right.
Difficulty index is the difference between the number of candidates who were able to solve a particular question correctly, divided by the total number of students.
Every question that is included in a cognitive assessment is validated via its difficulty index.
For example, if out of 100 students, 20 students were able to answer a particular question correctly, the difficulty index for that question would be: 20/100 = 0.5.
The difficulty level of a question is determined by its index range
Ideal difficulty index: In order to accurately filter the right talent for a particular job role, the difficulty index of each question included in the assessment should differ from each other in order to be able to differentiate between different candidates most accurately.
3)Discrimination Index of each question
The Fourth Step to creating an accurate aptitude test involves getting the Discrimination Index of each question included in the aptitude test right.
Discrimination Index is the power of a question to discriminate the students between those who scored high and those who scored low in a cognitive assessment for a particular job role . Discrimination Index is the basis of measuring the validity of a question. This index can be interpreted as an indication of the extent to which overall knowledge of the content area or mastery of the skills is related to the response on a question.
For example, if 100 people gave a cognitive assessment and 50 candidates scored above 50% and 50 candidates scored below 50%, then the discriminatory index for a specific question will be decided by the number of candidates in the upper 50 percentile who answered the questions correctly and the number of students in the lower 50 percentile who answered the question correctly. In this case, if 20 candidates from the upper percentile answered a specific question correctly, which was also answered correctly by 5 candidates from the lower percentile, the discriminatory index of that question would be:
D= (Ru-Rl)/(1/2 *T)
D= (20-5)/(1/2 *100 )= 0.3
D -discrimination index
Ru - the number in the upper group who answered the item correctly
Rl - the number in the lower group who answered the item correctly
T -the total number of both upper and lower groups
Ideal discriminatory index: The ideal discriminatory index for a question should be 0.4 and above. A cognitive assessment where each question has a discriminatory index of 0.4 or above, will be able to effectively filter our undesirable talent and select desirable talent accurately suited for a job role.
The Final Acid Test
The Fifth and Final Step in creating an accurate aptitude test involves tuning the aptitude level of the talent pool with that required in the job role.
Following the above three criterias, will ensure that the cognitive assessment is accurate in spotting the right talent for any job role.
However there is still something missing-
The cognitive assessment is aligned to meet the demands of the job role's cognitive skill requirement. But is it aligned with the talent pool's cognitive ability level ?
What I mean is-
Is the cognitive assessment suitable for the talent pool that is going to be giving the assessment?
The answer is - Not yet.
There are scenarios where our cognitive assessment/aptitude test can still fail. These scenarios are-
- The talent pool's cognitive ability surpass that required by the job for which they are being assessed. e.g going to an ivy league college to hire a data entry operator.
- The talent pool's cognitive ability is below that required by the job for which they are being assessed. e.g going to a community college to hire research scientists.
In these special circumstances, the only thing that can ensure that your cognitive assessment remains reliable and valid is experience, which we are proud to say, Mettl has plenty.
So how does Mettl battle these special scenarios and come out a winner?
The answer is - By taking into consideration the IQ level of the talent pool and adjusting the level of questions that the cognitive assessment has, to match them.
We have our own two special solutions to combat the two special scenarios:
- If the candidate pool possesses lower level of cognitive ability than required in the job role, the difficulty level of questions is reduced.
- Similarly, if the level of cognitive ability that the candidate pool possesses is higher than that demanded by the job role, the level of difficulty of the assessment questions is increase.
We use this framework (below) to decide the level of questions in a cognitive assessment/aptitude test, based on the level of cognitive abilities that the talent pool has and the job role requires.
Full disclosure, no company would be happy to take in compromised talent which scenario 1 provides us with.
And no candidate would like to settle in a job which is not a match for their intelligence (scenario 2).
But truth be told, both these scenarios exist and are very commonplace indeed!
Finding a talent pool that possess the exact cognitive ability as needed to perform a specific job role is rare indeed.
Till you find the perfect cognitive match for your job role, know that you are well taken care of and odds will be dealt with successfully by Mettl's 5 fold cognitive assessment reliability and validity check.
Stay Calm and carry on your hunt for top talent with Mettl by your side!
Topics: Aptitude Test