Giving feedback to a study participant is often a challenge to a researcher. To cope with this challenge, many short or automated feedback systems are used. Despite their convenience for the researcher, it is unclear if these instruments do not suffer from the Barnum effect – a tendency to perceive personality feedback as accurate, despite its being valid or invalid. “Your NEO Summary” is one of these instruments that provide quick feedback, so it is important to investigate its validity and determine if it is perceived as accurate not due to the Barnum effect.
The feedback sheet is a one-page summary used to provide a quick feedback for the NEO PI-R and NEO-FFI questionnaires. At the top of the page, there is a short introduction about the test used and about the summary sheet, and below there are 15 statements that describe all of the Big Five traits at different levels: highly expressed, averagely expressed, and lightly or not at all expressed. The person giving feedback just checks the boxes next to the statements adequate to the person being evaluated.
Creators of the NEO PI-R and NEO-FFI questionnaires have done some research to prove the validity of the “Your NEO Summary” and have concluded that the information provided in the summary is valid (Costa ir McCrae, 2012, p. 59). However, there remain some unanswered questions: since we were able to find just one study that provides information on the validity of the NEO summary sheet, more research on the topic is clearly needed; furthermore, despite the fact that the NEI PI-R and NEO-FFI questionnaires show the universality of the Big Five traits through cultures (McCrae, 2005; Schmitt, Allik, McCrae, & Benet-Martinez, 2007), there is some evidence to indicate a degree of cultural specificity of some of the Big Five traits (Allik, 2005; Triandis & Suh, 2002). Therefore, it is important to determine whether an instrument that accurately provides feedback to people in the USA can also provide accurate and meaningful feedback to Lithuanians.
The aim of this study was to determine whether “Your NEO Summary” and its component statements are valid. To realize this aim, three studies were carried out. In the first study, we tested whether a real summary sheet is perceived as more accurate than an inverted one.
In the first study, 269 university students filled in the NEO-FFI questionnaire and were promised feedback; they were informed that the aim of the study was to determine the validity of the feedback; 176 students (44 males, 132 females) came back to receive the feedback. Some of the participants got their real summary and some got an inverted one. The inversion was made so that the statement in the summary was as far from the truth as possible, so for a person whose personality T scores of a given trait were less than or equal to 50, we checked the statement in the summary sheet which indicated a highly expressed trait. If a person’s T score was above 50, we checked the statement indicating a low trait expression. After reading their personality feedback, participants were asked to turn over their sheets and answer the question printed on the back: they were asked to evaluate the accuracy of their feedback on a scale of 1 to 6, 1 being entirely inaccurate and 6 being entirely accurate. We found that the group that received the real (M = 5.04) “Your NEO Summary” perceived it as more accurate than did the group that received an inverted one (M = 4.31), and the effect was moderate (U = 2051.5, Z = –4.284, p < 0.001, r = 0.32).
In the second study, we analysed the relationship between the Big Five traits as measured by the NEO-FFI questionnaire, and the component statements of the “Your NEO Summary”. For this purpose, we constructed a form containing all of the feedback statements, making sure that no two statements describing the same trait were next to one another. These statements were also depersonalized. Each statement was evaluated on a seven-point Likert scale ranging from 1 (“describes me completely inaccurately”) to 7 (“describes me completely accurately”). University students participating in the study (N = 101, 38 males, 63 females) first filled in the NEO-FFI questionnaires and then the prepared form.
Correlational analysis determined that all but one of the feedback statements that indicated either high or low trait expression were correlated to the Big Five traits in the predicted direction: evaluations of the accuracy of statements describing the low expression were negatively correlated, and statements indicating a high expression were positively correlated with the Big Five traits. Some of the correlations were particularly high; for example, the correlation between the evaluation of the statement indicating a high extroversion was highly correlated with the trait of extroversion (rs = 0.775, p < 0.01). However, the evaluation of the statement indicating a high openness with experience did not correlate with the trait of openness to experience (rs = –0.063, p > 0.05).
Since the relationship between personality traits and the statements indicating the average trait expression is obviously non-linear, we conducted a repeated measure analysis to determine which of the three possible summary statements of a given trait is perceived as the most accurate in the group of people whose T scores of that trait were between 45 and 55. We found that only the statements indicating the average neuroticism and extroversion were evaluated as the most accurate; in all other cases, we found that statements of a high trait expression were evaluated as the most accurate.
In the third study, we repeated the procedure of the second study, but instead of the NEO-FFI questionnaire we used the NEO PI-R questionnaire; 82 university students (22 males, 60 females) participated in this study.
We found correlations similar to those described in the second study. However, openness to experience was found to be negatively correlated with the evaluation of the feedback statement indicating low openness to experience (rs = –0.226, p < 0.05) and the evaluation of the statement indicating a high openness to experience was not significantly correlated with openness to experience (rs = 0.215, p > 0.05). This is in contrast to what we found in the second study, and it might indicate that the NEO PI-R questionnaire items more accurately describe the behaviour presented in the statement. Also, the trait of openness to experience was found to be negatively correlated with the evaluation of the statement indicating high conscientiousness (rs = –0.359, p < 0.01).
In the cases of neuroticism, extroversion, and conscientiousness, the facets of the Big Five traits were found to be adequately correlated with the evaluation of the corresponding feedback statements; the correlations were fairly strong and in the predicted direction. However, some facets of the trait of agreeableness, namely straightforwardness (A2) and modesty (A5), did not significantly correlate with the evaluation of the feedback statement indicating high agreeableness. In the case of openness to experience, it was found that only one of its facets was correlated with the evaluation of the corresponding feedback statements, namely openness to actions (O4). The facet of openness to values (O6) was found only to be negatively correlated with the feedback statement indicating low openness to experience. Other facets of this trait showed no significant correlations with the corresponding feedback statements.
A repeated measure analysis showed that the participants whose T scores of a given trait were between 45 and 55 evaluated the feedback statement indicating an averagely expressed trait as the most accurate, the only exception being the trait of openness to experience, in which case the feedback statement indicating a highly expressed openness to experience was evaluated as the most accurate.
The results of the first study indicate that the “Your NEO Summary” is perceived as accurate not only because of the Barnum effect, but also because of its validity.
As found in the second and third studies, the correlations between the evaluation of the feedback statements and personality traits and their facets were mostly adequately high and in the predicted direction. These results, therefore, show that the “Your NEO Summary” is mostly valid, albeit there were some problems with the feedback statements indicating openness to experience.
The openness to experience scale in the Lithuanian translation of the NEO PI-R questionnaire shows quite good psychometric properties, so the predicted correlations should be found at least with this questionnaire; however it should be noted that openness to experience encompasses a lot of conceptually different facets (Griffin & Hesketh, 2004), and we might see a lack of correlations because of this property of the trait in question, or because the statements indicating openness to experience do not encompass the whole trait of openness to experience and describe only one facet of the trait, namely openness to actions.
Whatever the cause of the inconsistency, we recommend researchers to provide more detailed feedback about the trait of openness to experience and its facets. It is also noteworthy that some of the correlations between the evaluation of the feedback statements and the Big Five traits were quite high, so researchers should also provide more detailed feedback about these traits and their facets as well since the feedback statements provide information that is apparent and therefore non-informative.
We also noticed that those participants who had averagely expressed personality traits sometimes viewed feedback statements of apparent high social desirability to be more characteristic of them. This effect, however, was minimal.
This study had some limitations. Participants in all the studies were students, so research with a more representative sample is needed. The feedback inversion scheme used in the first study was quite simplistic, and further research should be done using the statements that describe more than three levels of trait expression. Also, the ratio of participants in the first study could also be considered a limitation.