Improvement of social functioning is an important treatment goal in mental health services and in intervention research (1). In adults with schizophrenia, the Personal and Social Performance Scale (PSP) is the most frequently used assessment instrument in research, and it has been evaluated as the social functioning measure with the highest quality (2). The PSP, developed by Morosini et al. (2000) (3), is a clinician-rated scale based on all available information concerning the patient’s social functioning. PSP provides comprehensive information on functioning as it includes both a total score of social functioning from 1–100 and scores on four subareas of functioning: A) Socially useful activities, including work and study; B) Personal and social relationships; C) Self-care; and D) Disturbing and aggressive behaviors (See visualization of the structure of the PSP scale in Figure 1). The PSP scale therefore allows for a detailed assessment of social functioning, recognizing that individual’s impairments may vary across areas, and ensures that functioning across these areas are considered systematically in the total score.
Structure and scoring guide of the Personal and Social Performance Scale
In research on early onset psychosis (EOP, onset < age 18 years), there is no clear consensus on the most appropriate social functioning measure. This implies that a wide range of different social functioning measures are used across studies, limiting the comparability of findings. The variety of instruments focusing on specific age-ranges further limits the comparability across age groups and complicates longitudinal research. Many commonly used instruments in adolescents, as well as adults, rely on a single global score from 1–100 and do not distinguish between impairments on different areas of social functioning. One study examining the validity and reliability of PSP in adolescents showed promising results (5). This provides interesting prospects as PSP potentially can comprise a high-quality instrument suitable to assess social functioning in samples both below and above age 18 years. In the current study, we contribute to the psychometric evaluation of PSP in adolescents by examining the inter-rater reliability based on a small sample of Danish adolescents.
The study sample consisted of eight young participants between age 13 and 17 years (sex-ratio 1:1). Four of the participants had a diagnosed psychiatric disorder and were recruited from the Child- and Adolescent Mental Health Center in the Capital Region of Denmark, while four participants had no known psychiatric disorder and were recruited through personal network of the research team. Healthy participants were included to ensure variation in social functional impairments across participants. All participants and/or legal caretakers (dependent on the participant’s age) gave written informed consent. A semi-structured interview on social functioning was conducted with the participants. The interview was age-adjusted to be suitable for adolescents, e.g. asking about school participation instead of employment. Interviews were audio-recorded and rated by six raters with clinical training as psychologists or MDs, including a chief psychiatrist. The raters were working in child- and adolescent mental health services and were trained in the PSP scale by an experienced researcher. The interviews were rated following the scoring guidelines outlined in Morosini et al. (2000), and raters were blinded from each other’s ratings. The PSP is rated by first rating impairments in the four subareas on a 6-point Likert scale from 1 ‘Absent’ to 6 ‘Very severe’. Then a 10-point interval of the total scale (from 1–100) is provided based on these ratings. The final total score within the 10-point interval is rated based on clinical evaluation. The absolute agreement between the six raters were evaluated with Intraclass Correlation Coefficient (ICC) using two-way random effects model for single measures and calculated for the following outcomes of PSP: the total score from 1–100; the 10-point interval; and the four subareas. There was no missing data across all measures.
ICC was calculated using IBM SPSS Statistics version 29.0.1.0.
The ICC values and range of ratings used by the raters are presented in Table 1. Agreement between the six raters on the eight interviews was good for both the PSP total score (ICC=0.85) and the 10-point intervals (ICC=0.80). Agreement on ratings of the four subareas varied. While the agreement was excellent (ICC=0.92) for subarea A (Socially useful activities), it was only moderate for the remaining three subareas B-D (Personal and social relationships; Self-care; and Disturbing and aggressive behaviors) with ICCs of 0.59–0.60. The range of ratings used by the six raters were narrower for subarea B-D than subarea A.
Inter-rater reliability for six raters on Personal and Social Performance Scale (PSP) using Intraclass Correlation Coefficient (ICC)
| PSP item (scale range) | Range of ratings used by raters | ICC value (95% CI) |
|---|---|---|
| Total score (1 – 100) | 39 – 90 | 0.85 (0.68–0.96) |
| 10-point intervals (1–10 to 91–100) | 31–40 – 81–90 | 0.80 (0.59–0.95) |
| Subarea A (1 – 6) | 1 – 5 | 0.92 (0.80–0.98) |
| Subarea B (1 – 6) | 1 – 3 | 0.59 (0.31–0.87) |
| Subarea C (1 – 6) | 1 – 4 | 0.60 (0.31–0.87) |
| Subarea D (1 – 6) | 1 – 2 | 0.60 (0.32–0.88) |
Adding to the limited literature on the usefulness of the Personal and Social Performance Scale (PSP) in adolescence, we explored the inter-rater reliability of the PSP between six raters in eight Danish adolescents. The study showed good agreement between raters on the PSP total score and excellent agreement on subarea A (Socially useful activities). Rating of impairments on the remaining three subareas was of moderate agreement.
The good inter-rater reliability of the PSP total score, when used in adolescents, is in line with the previous study by Ulloa et al. (2015) exploring the validity and reliability of PSP in adolescents aged 12–17 years (5). The study by Ulloa et al. (2015) found ICC values on the total score and on the four areas ranging from 0.82–0.99, indicating good to excellent inter-rater reliability. This is contrary to our results, showing only moderate ICCs for PSP subareas B-D. We suggest that the moderate ICCs for some of the PSP subareas in our study reflects both the low variability in impairments and the small sample size. These two factors can result in a low ICC, although the agreement between raters is high (6). The low variability was especially true for subarea B and D, where raters used only the ‘better functioning’ half of the 6-point Likert scale. On subarea A, there was the most variability with raters using five of the six scores, which might have resulted in the higher ICC for this sub-area. The study by Ulloa et al. (2015) reported mean scores of the subareas ranging from 3.1 to 4.3, corresponding to the middle part of the Likert scale, in their sample consisting of 40 young individuals with schizophrenia spectrum disorders. This indicates that their data was less skewed towards good social functioning compared to our study. Our inclusion of healthy participants has likely contributed to the low variability in social functional impairments. The moderate ICCs for the subscales B-D could also be a result of these specific areas being less applicable to the adolescent population compared to adult samples (for example regarding keeping a household), leading raters to evaluate impairments inconsistently. However, based on the range of ratings used by our raters (see Table 1), we suggest that the lower ICCs are primarily due to lack of variation in social functional impairments instead of lack of agreement.
The results of the current study should be cautiously interpreted considering the small sample and the low variability in impairments of some areas. The study sample consisted exclusively of Danish adolescents and the findings might not be generalizable to other countries. Despite these limitations, the good inter-rater reliability of the PSP total score across as many as six independent raters suggests that trained raters can reliably apply the PSP scale to adolescents. These findings together with the findings in the study by Ulloa et al. (2015) showing good reliability and validity of PSP, provides additional support for the application of PSP in adolescents.
Future research should explore inter-rater reliability in larger samples of adolescents and include adolescents with more variability in social functioning, for example various psychiatric diagnostic groups. This could provide a more accurate estimation of the inter-rater reliability and increase our understanding of the PSP’s performance across adolescent populations. Further psychometric evaluations are needed, for example exploring concurrent validity by comparing PSP ratings with ratings on an instrument on social functioning specifically designed and validated in adolescence.
This small-scale study supports the initial evidence concerning the appropriateness of applying the PSP to adolescent samples by showing good inter-rater reliability of the PSP total score. The inter-rater reliability was lower for some subareas, likely due to low variability in impairments in our small sample. Further studies with larger and more diverse samples are needed to provide insight into whether PSP could provide a comprehensive, high-quality assessment of social functioning in adolescence, which is highly needed in child- and adolescent mental health services. The possibility to follow PSP scores across the transition from adolescence into adulthood could improve quality of longitudinal research in social functioning.