Abstract:
Nothing in life is to be feared, it is only to be understood. Now is the time to understand more,
so that we may fear less.
Marie Curie
Debates about how best to test people from different contexts and backgrounds
continue to hold the spotlight of testing and assessment. In an effort to contribute to
the debates, the purpose of the study was to develop and evaluate the viability and
utility of nonverbal figural reasoning ability items that were developed based on
inspirations from African cultural artefacts such as African material prints, art,
decorations, beadwork, paintings, et cetera. The research was conducted in two
phases, with phase 1 focused on the development of the new items, while phase 2
was used to evaluate the new items. The aims of the study were to develop items
inspired by African art and cultural artefacts in order to measure general nonverbal
figural reasoning ability; to evaluate the viability of the items in terms of their
appropriateness in representing the African art and cultural artefacts, specifically to
determine the face and content validity of the items from a cultural perspective; and
to evaluate the utility of the items in terms of their psychometric properties.
These elements were investigated using the exploratory sequential mixed method
research design with quantitative embedded in phase 2. For sampling purposes, the
sequential mixed method sampling design and non-probability sampling strategies
were used, specifically the purposive and convenience sampling methods. The data
collection methods that were used included interviews with a cultural expert and
colour-blind person, open-ended questionnaires completed by school learners and
test administration to a group of 946 participants undergoing a sponsored basic
career-related training and guidance programme. Content analysis was used for the
qualitative data while statistical analysis mainly based on the Rasch model was
utilised for quantitative data.
The results of phase 1 were positive and provided support for further development of
the new items, and based on this feedback, 200 new items were developed. This
final pool of items was then used for phase 2 – the evaluation of the new items. The
v
statistical analysis of the new items indicated acceptable psychometric properties of
the general reasoning (“g” or fluid ability) construct. The item difficulty values (pvalues)
for the new items were determined using classical test theory (CTT) analysis
and ranged from 0.06 (most difficult item) to 0.91 (easiest item). Rasch analysis
showed that the new items were unidimensional and that they were adequately
targeted to the level of ability of the participants, although there were elements that
would need to be improved. The reliability of the new items was determined using
the Cronbach alpha reliability coefficient (α) and the person separation index (PSI),
and both methods indicated similar indices of internal consistency (α = 0.97; PSI =
0.96). Gender-related differential item functioning (DIF) was investigated, and the
majority of the new items did not indicate any significant differences between the
gender groups. Construct validity was determined from the relationship between the
new items and the Learning Potential Computerised Adaptive Test (LPCAT), which
uses traditional item formats to measure fluid ability. The correlation results for the
total score of the new items and the pre- and post-tests were 0.616 and 0.712
respectively. The new items were thus confirmed to be measuring fluid ability using
nonverbal figural reasoning ability items. Overall, the results were satisfactory in
indicating the viability and utility of the new items.
The main limitation of the research was that because the sample was not
representative of the South African population, there were limited for generalisation.
This led to a further limitation, namely that it was not possible to conduct important
analysis on DIF for various other subgroups. Further research has been
recommended to build on this initiative.
