Relationship-building skills are described as central in human service professions, and empathy and empathic skills are often emphasised as crucial ingredients of those skills (Bohlin & Eklund 2013). Education for professions such as doctors, psychologists, social workers, and teachers therefore includes courses and training in relationship-building skills. Many professional groups in the human services, however, have care and support functions rather than being specifically oriented towards treatment. Care and support professions, moreover, are often characterised by a violent working environment (Beiring et al. 2018; Nyberg et al. 2021), and client groups in, for example, psychiatric care, youth care, correctional facilities, and assisted living facilities are generally highly vulnerable. Thus, empathic skills and behaviour are also crucial here, but the level of education required varies.
Empathic skills are central in the concept of Motivational Interviewing (MI), as introduced by Miller and Rollnick (1991, 2013). MI has developed from a loosely structured intervention to an evidence-based style of conversation (Björk 2014). It was developed for the treatment of substance abuse but is also commonly used in several areas of social work (NBSH 2018). MI conversations take place between a therapist and a client to strengthen the client’s intrinsic motivation and ability to change. The therapist practices certain skills, such as empathy, compassion, and respect for the client, to help the client achieve progress (Miller & Rollnick 2013). Methodological fidelity to MI can be measured to ensure that the professional’s conversations are conducted in an evidence-based way (cf. Breitenstein, Robbins & Muennich Cowell 2012). An established instrument for measuring such methodological fidelity is Motivational Interviewing Treatment Integrity (MITI), where recorded MI conversations are coded based on parameters in MI: promoting change speech, reducing maintenance speech, empathy, and partnership. The first two are technical parameters, and the second two are relationship-building components. Moyers and Miller (2012) have suggested that for the ethical reason that treatment staff should not harm clients, workplaces should assess staff members’ empathic skills and ability to refrain from harmful behaviours, and that these capabilities should be investigated when recruiting new staff. In accordance with Moyers and Miller’s suggestion, the residential homes run by the Swedish National Board of Institutional Care (Statens institutionsstyrelse, SiS), which provides compulsory care for young people, decided to develop and trial a work test, in the form of a role play, based on the concept of MI. The aim was to develop and trial a new way of recruiting applicants for positions in the organisation, in which job applicants would be given an opportunity to demonstrate relationship-building skills in terms of empathy, collaboration, and non-authoritarian behaviour in a work test. In recent years, there have been several violent incidents between staff and young people at these residential homes (see e.g., Andersson & Andersson Vogel 2023; Barnrättsbyrån 2021; DN 2021; IVO 2023). Thus, SiS residential homes represent the type of human services organisations where professional support is often carried out in a context of violence or the potential for violence, and the level of education of the staff varies.
In the SiS project, the validated, well-established, and theoretically anchored instrument of MI is put into practise in a work test and thus, in a new and untested setting. In this sense, the project explicitly exemplifies a common but often implicit phenomenon in social work, that is, that methods and tools are tinkered with when they are used in practise. It also exemplifies what consequences such tinkering might have (cf. Skillmark & Oscarsson 2020). Moreover, the skill in focus in the SiS project – relationship building – includes the concept of empathy, which has multiple definitions and meanings (Gibbons 2011; Clark & Butler 2020). Studying the experiences of the SiS project can thus shed light on a complex skill at the core of social work and the challenges connected with assessing this skill. The aim of the present article, which is based on interviews with project stakeholders, is to add to the discussion on the balancing act required in evidence-based practise of maintaining psychometric stringency versus making contextual and situational adjustments. The focus is on exploring the possibilities for using and interpreting measures of MI in a new setting as well as the possibilities for assessing relationship-building skills in the recruitment process.
The SiS Project
SiS is the Swedish authority responsible for the compulsory care of adults with serious substance use problems and young people with extensive psychosocial problems or who have committed serious offences. The SiS project involved four special residential homes1 for young people that volunteered to use the work tests in all their recruitment processes.2 The authors of the present article were not involved in designing the project but were engaged to follow the project. The authors’ engagement resulted in a written report to SiS summarising the results from the research interviews. This engagement did not come with any type of financial compensation nor do the authors have any other financing from or connection to SiS. In the project, the two or three highest-ranked applicants in each employee recruitment process were asked to carry out an additional work test in form of a 20-minute role play. Applicants were informed that their own role was that of a school staff member (with counselling tasks), and that their task was to convince a pupil to return to school following a long period of absence after a conflict with a teacher. The pupil was played by an actor. The job applicants did not necessarily have experience or knowledge of working in a SiS institution. Regardless of education or job experience, however, everyone is familiar with the context of a primary school. This was the crucial reason for choosing this context for the test. The role play was conducted by phone and was audio-recorded. The conversation was then analysed and coded and scored based on core factors in the MI method. Although all four parameters of MI were measured in the project, the focus was on the relationship-building components: empathy and partnership. SiS decided to use the work test in the recruitment process for all positions in the four participating residential homes. The reason given for this decision was that the skills measured in the work test are important for the SiS organisation generally, and not only in the positions directly involved in treatment. The recruitment decision was informed by the results from the work test, together with ‘ordinary’ qualifications.
Coding was carried out by MIC Lab.3 In addition to recording and coding the work samples, MIC Lab was also involved in designing the case, providing actors, and giving verbal feedback to applicants. The coding results and written feedback were sent to the person responsible for recruitment at the institution and used as a basis for further assessment of applicants. Applicants also received brief verbal feedback from MIC Lab focussed on areas of improvement and demonstrated strengths. The audio-recorded work tests and the results/grades are not part of the data analysed in the present article.
Relationship-Building Skills in Social Work
Relationship-building skills, including empathy, have been central to the social work profession since its inception (King & Holosko 2012; Watson et al. 2022). Research consistently indicates that empathic practise positively influences client outcomes (Gerdes & Segal 2011; Moyers & Miller 2012). The concept of empathy, however, is multifaceted and lacks a cohesive theoretical foundation which has implications for both research and practise (Watson et al. 2022; Gibbons 2011; Clark & Butler 2020). Despite this, a review of empathy literature identified four consensus themes: understanding, feeling, sharing, and maintaining self-other differentiation (Håkansson Eklund & Summer Meranius 2021). These themes are not isolated but constitute a meaningful whole, although their overarching nature requires contextualization for both research and practical application. For example, investigations of the role of empathy in community policing (a profession that is comparable to coercive care in the sense that both involve meeting vulnerable clients and working in violent environments) suggest a conceptual model of empathy that includes affective response, self-other awareness, perspective taking, and emotion regulation, based on a review of research on the role of empathy in police work (Inzunza 2015a).
The SiS project discussed in this article employed relationship-building skills, including empathy, as defined by Miller and Moyers (2021). Miller and Moyers emphasise that expressing empathy involves communication and define accurate empathy as the process of ‘working to understand the other individual’s perspective and experience’ (Howard 2022, p. 74). They further present accurate empathy as a core therapeutic skill that involves seeing the world through the lens of the client, an approach inspired by Carl Rogers’ definition of empathy as ‘as if’ (Rogers, 1957, p. 99). In line with other researchers, Miller and Moyers (2021) stress that empathy can be seen as both an innate trait and a trainable skill (Howard 2022; Gerdes & Segal 2011). In accordance with the notion of relationship building abilities as partly innate capabilities, the testing of these capabilities in social work where the educational level of staff tends to be low seems valuable and useful.
The Role of Relationship-Building Skills in Social Work with Adolescents in Coercive Care
Research on social work with adolescents in coercive care emphasises relational building skills in a somewhat broader sense. Studies where adolescents in coercive care have been interviewed about their experiences have concluded that staff who have a care-based interaction style (based on understanding and mutual respect and with a participatory way of working) rather than a hierarchical and rule-based working style promote positive relationship building (Engström, Engström & Sellin 2020; Ungar & Ikeda 2017). Given the stressful working conditions and prevalence of violent situations in coercive care (Andersson 2021; Attar-Schwartz & Khoury-Kassabri 2015; Davidson-Arad & Golan 2007; Stenström & Pettersson 2021), initiatives to strengthen staff communication skills are essential for violence prevention. Previous research has shown a bidirectional relationship between stress and empathic skills, emphasising the need for further research and interventions in this area (Baby, Gale & Swain 2018; Fetter 2012; Lester 2010; Morphet et al. 2018; Testad et al. 2010).
Assessing Relationship-Building Skills in Recruitment
Despite the acknowledged importance of empathy in social work, methods for assessing relationship-building skills in recruitment are scarce. One exception is a Swedish support home that used the same procedure as the present SiS project on a smaller scale, with positive results (Kristiansson, Skogens & von Greiff 2022). Some self-reported instruments for measuring empathy skills in social work-related professions have been developed. Two examples are the Empathy Scale for Social Workers (King & Holosko 2012) and the Empathy Assessment Index (Lietz et al. 2011). Both of these scales are broadly theoretically anchored in empathy research based on the early psychodynamic theories and client-centred therapy, such as the work of Rogers (1951), as well as more recent contributions where empathy in the field of social cognitive neuroscience (SCN) is integrated with theories from the social sciences (Gerdes, Segal & Lietz 2010). The scales are suggested for use in educational contexts as well as self-evaluation and guidance for practitioners (Inzunza 2015b; King & Holosko 2012; Lietz et al. 2011). To the authors’ knowledge, however, they are not used in recruitment processes. A measurement method in the field of employee selection that might be the most comparable to the method used in the SiS project is the method of Situational Judgement Tests (SJTs). SJTs are measurement methods that present applicants with job-related situations and possible responses to these situations (Lievens, Peeters & Schollaert 2008). An SJT is designed to measure non-academic attributes, such as integrity, empathy, and team awareness (Patterson et al. 2012). For example, an SJT measuring empathy, integrity and resilience is used to select candidates applying to train as general medical practitioners in the UK (Patterson et al. 2009; Plint & Patterson 2010). The range of SJTs in social care is limited but a recent example is an SJT test developed for recruiting staff in home support for older people in England in order to promote person-centred care (Dunn, Tiffin & Wilberforce 2022).
Data and Methods
The heads of the four institutions (HIs) were first contacted to arrange short meetings informing them about the authors’ role: namely, to follow the implementation of a MI-based work test by analysing key persons’ experiences. The HIs were asked to describe the regular recruitment routines of their specific institution.4 The interviewees in this study consisted of (a) job applicants, (b) heads of institutions, departments, and human resources, (c) staff of SiS who were responsible for MI competence, and (d) an actor who was hired as a role play opponent in the MI conversations. The interviews had slightly different purposes (see Table 1).
Table 1
Overview of data.
| CATEGORY OF INTERVIEWEES (NUMBER OF INTERVIEWS) | METHOD | AIM |
|---|---|---|
| Staff responsible for strengthening MI competence at SiS and for developing the work test (n = 1) | Group interview via zoom (5 participants) | Description of expectations and design of the pilot project |
| Job seekers (n = 40) | Individual telephone interview | Experiences of and perception of work tests |
| Persons in the recruitment process; HI, HD and HR (n = 7) | Interview via zoom individually or in pairs (in total 9 IP) | Experiences of and perception of work tests |
| Actor MIC Lab (n = 1) | Individual telephone interview | Experiences of and perception of work tests |
Data collection
The interviews with job applicants were conducted by telephone; the other interviews were held via Zoom. All the interviews were audio-recorded. Separate interview guides were used for the different categories of interviewees. All staff interviews focussed on expectations and experiences of the recruitment procedure.
In the initial phase of the project, an approximately one-hour group interview with five participants was conducted with staff responsible for strengthening MI competence in SiS and responsible, together with MIC Lab, for formulating the work test. The five participants were all MI educators at SiS. The aim of the group interview was to gather information on the process of preparing the work test and to assess SiS’s expectations regarding the implementation of the work test in the short- and long-term (see Table 1). The interview guide contained overarching questions about what to include in the actor’s script for the work test, expectations for the work test (such as how it might improve professional work at SiS), and how the work test should be assessed in relation to other parts of the recruitment process. The interview was held via Zoom and was both video- and audio-recorded.
The process of recruiting applicants started with the actor, at the close of the role play, asking the job applicants for permission to provide their contact details to the researchers. It was explained that the researchers were not involved in the job selection process, and providing contact details did not imply agreement to participate in a research interview. Contact details for those who gave consent were e-mailed to the researchers, who then contacted the job applicants by phone to describe the purpose of the interview and inform them that participation was voluntary and anonymous. A verbal request to participate was made, and all those contacted gave their consent to participate. Some interviews were conducted in connection with this initial call, but the great majority took place at a later date. Most of the interviews were conducted two to five days after the work test. In six cases where contact details were provided to the researchers, interviews were not carried out.
Individual interviews were conducted with 40 applicants for different positions (see Table 1 and Appendix 1). The interview guide began with background questions about age, education (including MI education), professional experience, and the type of position applied for. The interview then moved on to discuss the applicant’s experience of and views on the work test. Concrete questions about information and preparation were interspersed with more comprehensive questions about competencies and skills that the job applicant was able to demonstrate or had wished to demonstrate in the work test. The interviewees (IP) were also given an opportunity to suggest areas of possible improvement regarding the form and content of the work test. The interviews lasted 15–30 minutes and were conducted in the autumn of 2021 and the spring of 2022. All interviews were conducted by phone and were audio-recorded.
Interviews with the HI, and in some cases the deputy HI, head of department (HD), or human resource manager (HR), were conducted in the spring of 2022. These interviews were aimed at highlighting expectations for the recruitment procedure and the work test and discussing experiences of using the test and the likelihood of continued use. Initially, the interviewees were asked to explain who received the coding results and how the results were interpreted. The interviews (n = 7) were conducted individually or in pairs and lasted 15–30 minutes. The interviews were conducted via Zoom and were audio-recorded.
The interview guide for the actor contained questions about the information the actor provided to the applicant, his perception of the quality of the conversations, his perception of the applicants’ ability to step into the role of school staff member, and the treatment he perceived himself as receiving in the role of ‘Jacob’. The interview was held by phone and was audio-recorded.
The task of interviewing the job applicants was divided between the authors. Both authors participated in all of the interviews with the HI/HD/HRs, the interview with the actor, and the group interview with those responsible for MI at SiS.
Analysis
All the audio-recorded interviews were transcribed verbatim. The transcription of the interviews with job applicants was generally carried out by the researcher who did not conduct the interview. In this way, both researchers gained good knowledge of the entire body of material early on in the analysis process. For the interviews where both authors participated, the transcriptions were distributed equally. The interviews were analysed thematically in a process inspired by Braun and Clarke (2006). After careful reading of the transcribed interviews, the analysis focussed on one professional group at a time. The researchers coded the interviews individually and then compared and discussed their coding. Relevant quotes from the interviews were then jointly entered into a coding scheme. The material was structured based on the overall questions that the interviews raised and was compiled in a condensed form into themes. The analysis, therefore, oscillated between coding at the individual level and themes at the overall level (cf. Braun & Clarke 2006).
The presentation of the results is structured according to the overall themes/questions that emerged and illustrated with quotes in italics. Quotes from different interview persons (IP) are separated by a semicolon, ‘;’. Job applicants are marked as J1, J2, etc. Heads of institutions, heads of departments and human resource managers are marked as HI/HD/HR 1, HI/HD/HR 2, and so on. Speakers in quotes from the group interview are not given an ID, since in this interview, it was the process rather than individual reflections that was in focus.
Ethics
This research project was sent to the Swedish Ethical Review Agency, which decided that the study does not fall under the jurisdiction of the Ethics Review Act, as no intervention with a research person or other intervention would be carried out in the manner specified in § 4 of the Ethics Review Act (dnr. 2021-03574). Ethical guidelines were followed for the job applicants, however: they were informed of the purpose of the study and that their participation was voluntary, anonymous, and not in any way connected to the recruitment process (see also above). The SiS staff was aware that full anonymity was not guaranteed. Their participation, however, was included in SiS organisation development work. Neither of the authors has any financing from or connection to SiS or MIC Lab.
Results
The results from the interviews are presented in the following order: staff responsible for strengthening MI competence at SiS, job applicants, heads of institutions, heads of department and human resource managers responsible for recruitment, and the actor.
Staff responsible for strengthening MI competence at SiS
The group members were initially asked to talk about the process of designing the content of the work test so that it would suit the specific context. The discussion among the group members mainly revolved around three thematic issues. The first concerned the intentions behind the work test. Three specific intentions were formulated:5 to test the applicant’s empathic skills, to avoid employing people who are obviously inadequate in the sense that they react aggressively or violently in provocative situations, and to produce synergistic effects on the work culture on the wards.
The second thematic issue concerns the work test’s relation to MI methodology. This was discussed in terms of potential consequences for the use and interpretation of the results. The group members all agreed that the professional skills measured by MITI are relevant to work in SiS institutions, but some questions needed to be addressed: Should the results be interpreted as a whole or are some parts more important than others? What are the implications of the varying levels of knowledge of MI among the tested applicants (educated in MI, well-trained in MI, or no experience of MI at all)?
The group agreed that certain parts of the MITI measures were more important than others:
…what we do not consider important is the ability to support change talk, to reduce maintenance talk. For our purposes, this is not important. It is a clear priority, so to speak, that the relationship-building parts in MI are what we want the test to measure.
The methodological and technical aspects of MI were described in deciding the level of difficulty of the work test. To allow for a meaningful distribution in the results, the test needed to be not too difficult while still picking up actions not consistent with MI:
…Two highly skilled MI trainers did not get full points [when trying out the test]. We would have wanted to see people getting fives all through if they are skilled. So, me and X (at MIC Lab) sat down afterwards and adjusted the instructions to the actor to reduce the difficulty levels somewhat.
The group expressed trust that the general skills that SiS requires would be exposed by the test, regardless of whether the tested person was trained in MI. It was stated that the test needed to capture curiosity and a person-centred attitude but also whether a person tended to nag or give advice.
The third issue dealt with how to achieve the desired effects. Several aspects of the design and conduct of the work test were discussed. It was stressed that the work test needed to discriminate in the ‘right’ way and that the actor must be given instructions about this.
…the actor needs to react differently depending on whether he encounters aggression, then he has to tighten and raise the dissonance a bit. But if he is received in a positive way, with understanding, and feels that the person is actually listening and taking him seriously, then the dissonance should decrease.
They consistently agreed that they preferred not to see MI-consistent behaviour and that job applicants offered employment had to score at least three on the relational variables. High scores, however, were not as important when it came to technical parameters. To be able to interpret the results of the work test, the group members stressed the importance of knowledge of MI among the persons responsible for recruitment and how the MITI evaluation was structured.
Job applicants
The work test was used in recruiting for all positions at the four institutions during the time the implementation process was followed. Among the 40 job applicants that consented to be interviewed, the great majority were applying for a position as a treatment assistant/pedagogue (n = 26).6 The rest were applying for positions as heads of department (n = 4), administrators of treatment, or positions not directly related to treatment (such as teachers) (n = 10). The purpose of these interviews was to elicit information about the applicants’ experiences and perceptions of the work test. Three overarching themes emerged: the case, possibilities to demonstrate abilities, and improvements. For background information (gender, age, education and MI education) on each interviewee, see Appendix 1.
With a few exceptions, the job applicants were positive towards the work test. A majority highlighted the actor as crucial, saying he made them forget that they were talking to an actor: ‘I believed that it was Jacob, age 15, he was damn good!’ (J35); ‘it worked surprisingly well’ (J2). Some described difficulties in ‘playing’ a school employee meeting a pupil: ‘Really strange, unnatural to handle the part of a school staff member’ (J16). Some stressed that the actor ‘could have been more upset and obstructive’ (J39) and that it was ‘too peaceful and not really realistic’ (J18). Those seeking non-treatment-related posts sometimes questioned the relevance of the test to the position applied for.
All but a few job applicants stressed how the work test gave them an opportunity to demonstrate their abilities and competences. When asked to describe which abilities and competences they had the chance to demonstrate, they mentioned skills relating to behaviour: ‘dedication, responsiveness empathy and being secure’ (J5). They also mentioned skills relating to methodology: ‘low arousal approaches, being solution-centered’ (J7); ‘the role of resistance’ (J22); ‘confirm and cooperate’ (J24); ‘asking open questions and establishing partnerships’ (J17). Not surprisingly, terms connected with MI dominated for those educated in MI, even though some IPs emphasised that MI is ‘common sense’ and that they did not think about using or not using MI. A few people did not perceive that the work test gave them an opportunity to show their abilities at work, referring to the school context, the lack of body language [on the telephone], or the fact that such conversations were not relevant to the (non-treatment-related) job being applied for. Others said that they were uncomfortable with the situation or with role playing, or that ‘he [Jacob] acted too nice’ (J16), thus they were not provided opportunities to demonstrate solution-centred competences. One IP commented: ‘In reality, kids don’t always have this kind of vocabulary and willingness to take part in a conversation’ (J1).
When discussing suggestions for improving the work test, context and lack of body language were also mentioned. Body language included both that of the IP and that of Jacob: ‘seeing body language is a big part of motivational work’ (J5). Comments on context had to do with the fact that the professional role play was not in line with the position being applied for: not ‘the kind of conflict that appears with youth at SiS’ (J12). The most common suggestion for improvement concerned time, although the comments were slightly contradictory. Some suggested more time for rounding off and learning more from the conversation, while others stressed that the length of the conversation was unrealistic: ‘Talking for 20 minutes with a 15-year-old boy—in reality you’ve got maybe one or two minutes. Most kids don’t want to talk that long’ (J18). There was also criticism of MI relevance:
[I] don’t think the conversation gave an opportunity to show this kind of competence. It is not realistic. You have to first build up some sort of relation of trust, then you can start talking about change. The person in the case hasn’t expressed any targeted behaviour change so I see a technical problem there for coding the conversation (J18).
Some highlighted the importance of clarifying that the conversation would be MI-coded. One person was sceptical about awarding points for this kind of work test since it might lead to a focus on ‘scoring points’:
[it] risks becoming technical when you know about MI and coding…the conversation might be stiff and you could lose the listening part. The scores do not tell you much, you have to listen to the conversation (J1).
The dominant view, however, was that the work test could give the employer relevant and robust information on the job applicants’ qualifications and behavioural skills: ‘Everyone can get an education but not everyone might be suitable’ (J5); ‘[it’s] good to check this kind of ability. To sort out those who are not suitable’ (J10). Some also highlighted how the work test gave the applicant an opportunity to reflect on their own suitability for the job: ‘Good thing to think about, “Do I really want to do this?” It’s a test for handling difficult conversations—for both the employer and the job applicant’ (J15). It also gave an indication of what the employer considered important for the job: ‘It makes me feel safe that other newly employed people have also done this test and proved their ability in this sense as this is what it’s all about’ (J31). Many of the applicants emphasised that the test should be a complement to other qualifications. Some of those applying for management jobs suggested that the test might be relevant for jobs ‘on the floor’ but maybe not for higher-up positions where education and work experience would be more important. Others in this group considered it important for the employer to see how someone in a more senior position would act in this kind of situation.
Head of institutions, heads of departments, and human resource managers responsible for recruitment
Nine persons responsible for recruitment at the four institutions were interviewed: five individually and four in pairs, making a total of seven interviews (see Appendix 1). The interview guide contained questions both on their expectations for the work test as an addition to their regular recruitments procedure and their experiences of using it over approximately a half-year period. The aim was to collect information about how the work test was used, possible obstacles to its implementation, and how these were handled.
Expectations for the work test were that it would be an additional tool for avoiding bad recruitment:
To capture people with a talent for active listening and a confirming and sensitive communication style…and to avoid recruiting someone who has absolutely no talent for this, who would have a hard time working with us in the way we want them to. If we keep using this systematically, I think that we will nourish this culture, and that becomes a signal to the whole organization…of what you need to be good at (HI1).
In relation to assessment of the work test, an ideal picture versus reality emerged. The ideal picture was: ‘The first thing we got from the manual was that those with scores of one or two should not be hired since these people do not get better at partnership and empathy’ (HD6). During the interviews, however, it became clear that the HI/HD/HRs did have some room for manoeuvre that permitted individual assessment of the importance of the different components of the work test. The following describes the recruitment of a candidate known by the HI/HD/HR, who had a positive experience of that person but where the candidate’s test results were not as good as those of the competing candidate:
X [job applicants name] didn’t have as good results in the MIC Lab test as the other candidate but then you have to consider other factors in the evaluation. It’s not just about a confirming and listening communication style, there are other things, since I actually know both of them. Then I have to say that I do not employ the one with the best results in the work test. You have to be reasonable about the impact of the test (HD1).
A majority described deviating from the instructions, mostly justified by and discussed in relation to a knowledge of MI: ‘I’ve hired people with ones and twos who have turned out to be fantastic with our youngsters with really nice behavior and large amounts of empathy, but in the tests, they got low marks because they were unfamiliar with the methodological technique’ (HD6). Interpretation of the numeric scores in the test results coded by MIC Lab as opposed to the overall picture was constantly problematized: ‘…if we only look at the results then there are some who should not work with people at all if they have as low results as ones and twos (HD6); but then you have to consider the personal impression as well’ (HD5b).
The results of the work test are presented in several ways: as scores and as written comments both on aspects for further improvement and on the recorded conversation. Awareness of these forms and knowledge of how to interpret them varied. Those who were aware of the different forms of results, and had some knowledge of how to interpret them, highlighted the importance of not just using the scores but considering all aspects of the results in order to interpret the test:
I can always go to the webpage—I didn’t know that from the start—and listen to the conversation. So far, I have mostly looked at the scores but I discovered later that there are also written reflections on what can be improved—what can be developed and done better—and this strengthens the quality [of the assessment] (HR4a).
The advantage of listening to the conversation was related to the possibility of making a judgement that took account of (the lack of) MI competence:
We get the recorded conversation as well now—and this is very important for how to classify this from an MI perspective. If there are low scores, we need to listen to see the basic behavior. Because there are some things that can be corrected with little effort (HI3); one can see a huge difference, if you have studied MI before you can get high scores, but if you haven’t studied MI then you can have a really nice conversation but basic mistakes will generate a low score (HD6).
Actor
The actor emphasised that while he lacked in-depth knowledge of coding, he understood that minor details could significantly impact the quality of a conversation. In contrast to previous recordings the actor had done for MIC Lab, where the other participant was trained in MI, this time there were job applicants participating in the work test who were unfamiliar with how to engage in the conversations. Instead of focussing on MI components, they approached the conversation in their usual working manner. Consequently, despite displaying responsive and positive behaviour, they missed out on MI points:
They were responsive and engaged with me in a way that felt good but anyhow, those small things, such as affirming my ideas or maybe asking a question instead of arguing with me…they were very nice but they walked straight into traps…which they needn’t have done if someone had said two or three sentences about how to behave to get high scores.
The actor noted their pleasant interactions but highlighted missed opportunities, such as affirming ideas or posing questions instead of engaging in arguments. He illustrated this with an example of a conversation with an empathetic woman with whom he established good contact, but who did not adhere to the MI protocol, potentially affecting her score. Despite wanting to convey his feelings during the role play, he acknowledged that it wasn’t his task. In other instances, he observed individuals with the right intuition whose statements went awry in the given situation, highlighting the perceived rigidity of the test. The actor emphasised the challenge of balancing the need for adherence to MI parameters with the nuances of genuine empathetic interactions.
Discussion
The outcome of incorporating the work test into the recruitment process remains unknown, and the present qualitative data does not allow for a conclusive discussion on its efficacy. Instead, the discussion here focuses on the concept of ‘tinkering’: striking a balance between maintaining psychometric stringency and making contextual and situational adjustments in evidence-based practise. The initial interview with staff responsible for strengthening MI competence at SIS identified three specific intentions with the work test: ‘to test the job applicant’s empathic ability, to avoid employing people who are directly unsuitable and to influence the work culture in SiS institutions’. These intentions were confirmed in the HI//HD/HR interviews.
The predominant view among those who completed the work test was that it provided an opportunity to demonstrate their relationship-building ability. Descriptions touched on relationship building both in general terms and, for those educated in MI, in the specific context of the MI method. However, a challenge emerged in coding according to MI Treatment Integrity (MITI), which primarily measures fidelity to the MI method aimed at enhancing clients’ intrinsic motivation and ability to change. While the work test’s stated purpose was to assess general relationship-building skills, the extent to which it captured these skills versus specific MI fidelity remained unclear. This ambiguity was reflected in the group interview with staff responsible for strengthening MI competence at SiS, where members expressed confidence in the test’s ability to capture general skills but acknowledged placing varying importance on the different components of the results.
On the one hand, discussions in the group interview reflect the fact that the work test might examine relationship-building skills in a broader sense, which has been pointed out as important in social work in general and specifically in social work with adolescents in coercive care (Engström, Engström & Sellin 2020; Ungar & Ikeda 2017). On the other hand, the discussion points out a lack of clarity about what is captured in the test and how innate traits or trained skills (for those who are educated in the MI method) can be balanced and can affect the results. Even though the staff responsible for strengthening MI competence at SiS stress that the most important test parameters are the ones that measure relationship building (empathy and partnership), the test also measures more technical MI parameters (promoting change speech, reducing maintenance speech).
Concerning the intention to avoid hiring people unsuitable for provocative work situations, stakeholders at SiS agreed on its importance, but a challenge arose in assessing how the work test aligned with this goal. While MI-consistent behaviours could be measured, the test did not evaluate how applicants would behave in emotionally charged conflict situations. Here, the SiS project rests rather on a hope that a demonstration of empathic ability in a less provocative situation, assessed on the basis of an applicant’s ability to work on strengthening a client’s drive and ability to change, can also predict how the applicant would act in more tense situations. There is nothing in the results that contradicts this hope, but the relationship between an empathic approach and the ability to handle aggressive and tense situations needs to be investigated further. Previous research suggests that such a connection is possible (Wagaman et al. 2015). For example, the importance of strengthening staff communication skills has been raised as part of violence prevention work. However, the need for more research in this area has been emphasised (see e.g., Baby, Gale & Swain 2018; Morphet et al. 2018; Norman et al. 2020).
Regarding the third intention, to influence the work culture, job applicants perceived the work test as a signal of what the employer deemed important. While this perception is suggestive of the test’s potential influence on work culture, conclusive evidence was lacking.
Overall, the results indicate that the work test could have the potential to be used for the purposes defined by SiS. As discussed above, it is possible that the conversation in the test can provide evidence of general relationship-building skills, but the extent to which this is distorted by more specific knowledge of MI methodology and practise is still unclear. However, it does appear that relatively detailed knowledge of what MITI measures and how the results should be interpreted is required to assess and use the results. Several implementation issues therefore need to be addressed, notably how to organise the implementation of the work test and how to review and organise the interpretation and use of the results.
The results show that a balancing act is taking place. On the one hand, the aim is a measurable result using an established, validated, evidence-based method. There is strong agreement that empathy, which is one of the things being measured, is important in the work for which the applicants are being recruited. Furthermore, work tests have proved an effective way of evaluating competence (Lindelöw 2016; Schmidt & Hunter 1998). On the other hand, it is difficult to find ways to evaluate empathic behaviour. There are also biases and shortcomings in how well the method relates to the specific situation in which it is being used, which is reflected in the results. The way in which the balancing act is handled can be compared to the handling of ‘fluid objects’ (de Laet & Mol 2000; Law & Mol 2001; Law & Singleton 2005), in that there is usually an adaptation to the specific situation when standardised tools are used in practise. A possible alternative to the present work test would be the use of an SJT, developed for the specific job-relevant situation. An SJT might offer an objective way of reliably assessing the requested attributes (Patterson et al. 2012). However, the present work test has the advantage of being based on conceptually clear variables that are possible to entangle in further use.
Implementation challenges were underscored, emphasising the need for well-executed implementation to bridge the gap between the expectations from the staff responsible for MI competence at SIS and the reality of the SiS project. Job applicants’ satisfaction hinged on the actor’s skill in creating authenticity, highlighting the importance of an authentic experience in demonstrating abilities.
These results suggest that the work test should be used to complement other assessments or qualifications. In the interview with the MI group, a detailed discussion was held about what different test results from the MI coding might mean, which was far beyond the knowledge of most recruiters. Several of those involved in the recruitment process emphasised the need for some room for manoeuvre in interpreting the work test. This, however, would require knowledge of MI and how the MITI results should be used: for example, whether the scores are used to rank or as a cut-off, or whether limits should be established or a minimum score required in order to obtain employment. The need for room for manoeuvre also includes deciding when in the process the work test should be taken and how the test should be interpreted for those who have MI training versus those who do not. As the results reveal, access to documentation, such as coding scores, written comments on the results and audio recording, varied among those involved in the recruitment process, which affected their ability to make sound decisions. There is also a need for competence in interpreting different forms of documentation, which the results show also varied among the people involved in the recruitment process. Background, room for manoeuvre, and competence are thus three central aspects to be discussed when implementing this type of work test. While the staff responsible for strengthening MI competence at SiS emphasised that knowledge of MI and MITI is necessary for recruiters, and that they perceived that they were training them in this regard, the results show that this was not the case even in the small format of the SiS-project. This emphasises the importance of well-executed implementation, which is a recurring theme in implementation research (see e.g., Meyers, Durlak & Wandersman 2012).
Limitations
Even though we emphasised that the interviews were independent of SiS, the employment selection procedure, and MIC Lab, and that participation was anonymous, the applicants were in a dependent position, and this might have influenced how they described their experiences of the work test. It could also be the case that the applicants we did not get in touch with had different attitudes about the work test. It is also worth noting that an overwhelming majority of the research interviews were conducted before the applicant received feedback from MIC Lab.
Overall, the results imply that the work test was generally appreciated by all the parties involved and appears to have the potential to become a complementary tool in recruitment. The results also highlight the importance of solid implementation work. One challenge regarding the specific work test described here is its clear connection to MI and coding according to MITI, and the extent to which it is about treatment in general or related to MI. Within the framework of the SiS project, it was not possible to study possible improvements in work culture or the working environment for employees, or in institutional care for young people. Addressing these questions will be very important in future.
Conclusions
The SiS project study provides insights into the challenges of applying a psychometrically sound and validated method, emphasising the need for detailed discussions on these challenges. The results suggest that tinkering is often necessary when using standardised measures for complex skills like relationship building. Advanced knowledge of the method, measurement, and organisational context is crucial. While the study cannot provide definitive answers, it highlights the importance of addressing obstacles and difficulties in such situations. The study proposes that a nuanced understanding of the method, measurement, and organisational context is essential even with well-established methods. Further exploration is needed to understand the impact of the work test on work culture and its potential improvements.
Additional File
The additional file for this article can be found as follows:
Notes
[1] The four participating residential homes each have the capacity to care for 30–50 young people and have 100–200 fulltime employees. In this text, the terms residential homes and institutions will be used synonymously and refer to the residential homes run by SiS.
[2] Staff (including interviewed HI/HD/HR) at the four participating residential homes were not involved in the process of developing the work test.
[3] MIC Lab is a company providing MI training, manual based coding of recorded conversations and feed-back according to MI (miclab.se).
[4] These informational meetings served basically as information about how the institutions were organized and were not audio-recorded since they were not part of the analysis.
[5] The same intentions were expressed during preparation meetings held with the HI. These meeting were not audio recorded, however, and nor were they part of the actual data collection; thus they are not included in the results.
[6] Nineteen women and seven men. For other groups, gender and age are not reported for ethical reasons as the number of interviewees is significantly lower. The age range was 21–58 years (m = 37). Six IPs had a university degree in social sciences (Bachelor’s); 14 IPs had a two-year qualification in applied sciences; and six IPs had only a high school diploma. Twenty-eight (70%) had some form of MI training.
Acknowledgement
This article is partly financed by the Swedish Research Council for Health, Working Life and Welfare (FORTE) within the research programme ‘The future of social service work with substance use (FUSS): clients, trajectories, organisations and collaboration in a changing treatment system’ (dnr 2023-01746).
Competing Interests
The authors have no competing interests to declare.