Figure 1
Transformation Youth Care in North-Veluwe region: make use of the strength of the ‘Civil Society, invest in prevention and integrate prevention and care to lower costs for intensive youth care (Benjamins et al, 2015).
Figure 2
Logic model of change for EPR-Youth, based on the WK Kellogg Foundation logic model development guide.
Table 1
Overview of chosen process indicators and implementation outcomes, corresponding data-collection method, examination period, and target group for the process evaluation of EPR-Youth.
| INDICATOR/OUTCOME: DESCRIPTION | DATA | PERIOD | TARGET GROUP/ACTORS |
|---|---|---|---|
| Delivery process: Has EPR-Youth been developed as intended and in accordance with the contract? | Contract between IT-developer and CJG-organizations Document ‘system assessment’ Project documentation | Sept ’19 (immediately after implementation) | Not specified |
| Implementation process and context: Analysis of legal/ethical, socio-cultural, geographical political and socio-economic contextual aspects that were affecting the implementation process. | Verbatim transcripts project group meetings (34 × 3 hours), steering committee meetings (17 × 1,5 hours) and consultative group meetings (7 × 2 hours) | May ’18–Sept ‘19 | Steering committee, project group, consultative group parents & adolescents |
| Semi-structured focus group interviews with steering committee (n = 6) and project group (n = 8), 1,5 hours each. | June ‘20 | Steering committee and project group | |
| Project documentation | Jan ’16–Sept ‘19 | ||
| Acceptability: To what extent were users, both professionals and clients, satisfied with the intervention? [23] | Professionals’ questionnaire (n = 66): experienced ease-of-use and experienced usefulness | Feb ’20 | CJG-professionals |
| Questionnaires parents (n = 914) and adolescents (n = 89): desirability client-access, actual access, and experienced ease-of-use. | Sept–Nov ‘20 | Parents and adolescents that visit a CJG | |
| Semi-structured focus group interviews with professionals (n = 12), parents (n = 8) and adolescents (n = 4): how do professionals and clients experience the use of EPR- Youth and the client-portal? | Nov ‘20 | Professionals and clients | |
| Adoption: To what extent were clients using the client-portal? [23] | System data: monthly and total number of clients that logged on to the portal | Sept ’19–Dec ‘20 | Clients |
| Questionnaires parents (n = 914) and adolescents (n = 89): percentage of respondents that logged on to the client-portal | Sept–Nov ‘20 | Parents and adolescents that visit a CJG | |
| Appropriateness: To what extent does EPR-Youth match with working processes of professionals? [23] | Meeting reports of project group and steering committee | May ’18–Sept ‘19 | Steering committee and project group |
| Semi-structured focus group interviews with professionals (n = 12): Do professionals feel a match between EPR-Youth and their working processes? | Nov ‘20 | CJG-professionals | |
| Fidelity: To what extent are professionals and clients using EPR-Youth and the client-portal as intended, in accordance with the vision on transformation? [23] | Semi-structured focus group interviews with steering committee (n = 6) and project group (n = 8), 1,5 hours each. | June ‘20 | Steering committee and project group |
| Semi-structured focus group interviews (1,5 hours) with professionals (n = 12), parents (n = 8) and adolescents (n = 4): Do participants experience that EPR-Youth is supporting professionals to work in accordance with the vision on transformation? | Nov ‘20 | Professionals and clients |
Figure 3
Timeline for development and implementation of EPR-Youth.
Table 2
Summary of qualitative themes and sample quotations.
| THEME | RESPONDENT | SETTING | QUOTE |
|---|---|---|---|
| A1: Implementation process, barriers | |||
| Complexity of co-creation | Project leader | Project group meeting | We are going to use one EPR-system with three organizations, meaning we’re going to merge virtually. I think we underestimated the complexity of that. |
| Preventive health worker | Focus group project members | We have had a great deal of trouble pretending to be one organization, while we are not at all…. We are not working from one vision. | |
| Lack of leadership | IT worker | Focus group project members | A plan of action was lacking, as well as clarity about everyone’s role. |
| Manager | Focus group steering committee | The steering committee, including myself, too easily assumed that everyone knew the ultimate objective. | |
| Preventive health worker | Focus group project members | The project leader kept saying that we had to ask the steering committee about issues. | |
| Concerns about legal aspects | Father | Consultative group of clients | As a parent, I really don’t want that my child’s medical letters can be downloaded to my computer without warning. That feels unsafe. |
| Preventive health worker | Project group meeting | Professionals must discuss with an adolescent whether their parents are allowed to access their record, but they must speak with the parents as well. | |
| Youth care worker | Project group meeting | Maybe more medical information should be shown in the general record because some of that is really relevant for youth care workers. | |
| Adolescent | Focus group clients | As a child, especially when you grow up, you want to decide who can access your record and who can read it. | |
| Lack of information | Preventive health worker | Focus group professionals | A year ago, information has been added to the invitation letter parents of newborns received, but we did not inform parents of older children. |
| Mother | Focus group clients | I think you should really explain the client portal… to every parent. Don’t assume that they will find the client portal by themselves. | |
| A2: Implementation process, facilitators | |||
| Structuring the process | Youth care worker | Focus group project members | The deadline when EPR-Youth was delivered had to be met at all costs. Because the deadline was clear we were able to make rigorous decisions. |
| Preventive health worker | Focus group project members | This is what people had been suggesting for so long: one group should be working on the content, and another on the newsletter, the user’s manual etc | |
| Clarifying the vision | Preventive health worker | Focus group project members | Once everyone clearly understood the ‘why’, decisions were quickly made. |
| Medical specialist | Focus group steering committee | Now that professionals begin to understand the vision, the resistance among them has considerably decreased. | |
| Pioneering spirit | Youth care worker | Focus group project members | Having taken up this challenge together that no one had tackled before made the difficult process more tolerable. |
| Youth care worker | Focus group project members | It’s nice to tell that this project is unique in the Netherlands. | |
| Legal aspects | Medical specialist | Project group meeting | From July 2020, everybody will be legally obliged to provide digital access to health records. |
| Project leader | Project group meeting | In my opinion a thorough legal assessment framework has been built already, and a legal guide has been provided. | |
| B: Implementation outcomes | |||
| Acceptability | IT-worker | Focus group project members | We have had relatively few start-up problems. Professionals ask practical questions about the record, but nothing like “what a miserable system” or “we can’t work with this” |
| Preventive health worker | Focus group professionals | I really like this system, EPR-Youth is easy to fill in | |
| Adolescent | Focus group clients | When you enter the website you need to verify in four steps. It really takes time before you are able to read things. | |
| Adoption | Adolescent | Focus group clients | Somebody asked if we wanted to complete a questionnaire and then we read about EPR-Youth. Otherwise, we would not have known. |
| Appropriateness | Youth care worker | Focus group professionals | As a youth care professional, it feels as if I am a visitor that is ‘allowed’ to report in a preventive healthcare record. |
| Youth care worker | Focus group professionals | Sometimes I think how I am going to report all this, because I feel a 12-year-old should not have to read that. | |
| Fidelity | Preventive health worker | Focus group professionals | When a child starts at school, you don’t have to transfer the record to the school doctor, because they continue working in the same system. |
| Manager | Focus group steering committee | With EPR-Youth it is no longer possible to report things that clients are unaware of. | |
| Youth care worker | Focus group professionals | I have a lot of colleagues who still write their referrals in MS Word. | |
Figure 4
Clients’ and professionals’ user experiences with EPR-Youth. Respondents scored on a 5-point Likert scale, ranging from ‘absolutely’ (1) to ‘not at all’ (5) for clients, and ranging from ‘totally agree (1)’ to ‘totally disagree’ (5) for professionals. Scores were reversed for all questions, except the ones marked with an asterisk, resulting in low scores representing a negative opinion and high scores representing a positive opinion. In the client questionnaire, Q2–4 were answered only by parents and adolescents that had accessed the client-portal.
Table 3
Demographic characteristics of portal users (from system data), compared with general population of the North Veluwe region, aged 15 to 65. Source: Dutch Central Bureau for Statistics.
| TOTAL | NUMBER OF USERS (PERCENTAGE) 5174 | PERCENTAGE AMONG INHABITANTS NORTH- VELUWE, AGED 15–65 YEARS | |
|---|---|---|---|
| Sexa | Male | 659 (12.7) | 50.5 |
| Female | 4509 (87.1) | 49.5 | |
| Unknown | 6 (0.1) | ||
| Ageb | 15–29 years | 1107 (21.4) | 28.4 |
| 30–44 years | 3964 (76.6) | 27.6 | |
| 45–64 years | 102 (2.0) | 44.0 | |
| Native countryc | Netherlands | 4895 (94.6) | 92.4 |
| Surinam | 5 (0.1) | 0.2 | |
| Netherlands Antilles | 5 (0.1) | 0.1 | |
| Turkey | 28 (0.5) | 1.0 | |
| Morocco | 15 (0.3) | 0.5 | |
| Europe, North America, Oceania, Indonesia, Japan | 105 (2.0) | 2.8 | |
| Other country | 121 (2.3) | 3.1 |
[i] a Significantly different from North Veluwe population (Chi2-test (1) 2945.942, 2-sided p < 0.001).
b Significantly different from North Veluwe population (Chi2-test (2) 6671.353, 2-sided p < 0.001).
c Significantly different from North Veluwe population (Chi2-test (6) 41.644, 2-sided p < 0.001).
Table 4
Comparison of login percentages (between brackets) among respondents of the client questionnaire, according to socio-demographic characteristics.
| PARENTS (n = 911) | ADOLESCENTS (n = 87) | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| LOGGED IN n = 490 (53.8%) | NO PORTAL USE n = 421 (46.2%) | STATISTIC | 2-SIDED p-VALUE | LOGGED IN n = 14 (16.1%) | NO PORTAL USE n = 73 (83.9%) | STATISTIC | 2-SIDED p-VALUE | |||||
| Educational level | ||||||||||||
| Low | 32 | (50.8) | 31 | (49.2) | χ2 = (2) 9.285 | 0.01 | 6 | (15.0) | 34 | (85.0) | χ2 = (1) 0.671 | 0.41 |
| Middle | 207 | (50) | 207 | (50) | NA | NA | ||||||
| High | 230 | (60.5) | 150 | (39.5) | 7 | (22.6) | 24 | (77.4) | ||||
| Missing | 21 | 33 | 1 | 15 | ||||||||
| Family composition | ||||||||||||
| 2-Parent family | 452 | (58.9) | 284 | (41.1) | χ2 = (1) 49.523 | <0.001 | 5 | (16.1) | 26 | (83.9) | Fisher’s exact | <0.001 |
| Other situation | 16 | (18.8) | 69 | (81.2) | 7 | (15.9) | 37 | (84.1) | ||||
| Missing | 22 | 36 | 2 | 10 | ||||||||
| Native country | ||||||||||||
| Netherlands | 431 | (54.5) | 360 | (45.5) | χ2 = (1) 0.447 | 0.504 | 12 | (17.4) | 57 | (82.6) | Fisher’s exact | 0.58 |
| Other country | 15 | (48.4) | 16 | (51.6) | 0 | (0.0) | 6 | (100.0) | ||||
| Missing | 44 | 45 | 2 | 10 | ||||||||
| Age children | ||||||||||||
| 0–3 y | 401 | (77.0) | 120 | (23.0) | χ2 = (1) 264.065 | <0.001 | NA | |||||
| 4–18 y | 67 | (20.2) | 265 | (79.8) | NA | |||||||
| Missing | 22 | 36 | NA | |||||||||