A Comparative Case Control Study between Innovations Team Malta and Innovations Team UK Working with Hard to Reach Young People with Complex Mental Disorders
Article Information
Andrea Saliba*, Nigel Camilleri
Department of Psychiatry, University of Malta, Msida, Malta
*Corresponding Author: Andrea Saliba, Department of Psychiatry, University of Malta, Child and Adolescent Psychiatrist, Mental Health Services Malta, Mount Carmel Hospital, Triq L-Imdina, Attard ATD 9033, Malta
Received: 14 April 2022; Accepted: 22 April 2022; Published: 02 May 2022
Citation: Andrea Saliba, Nigel Camilleri. A Comparative Case Control Study between Innovations Team Malta and Innovations Team UK Working with Hard to Reach Young People with Complex Mental Disorders. Journal of Pediatrics, Perinatology and Child Health 6 (2022): 233-244.
View / Download Pdf Share at FacebookAbstract
Background: Hard-to-reach young people (HTRYP) can be defined as YP who are suffering from complex mental disorders who are also at risk, disadvantaged, marginalized and who slip through the healthcare system or are unwilling to engage with mental health services.
Method: This case control study compared the outcomes from two HTRYP services, (Innovation Team (IT) Malta and Innovation Team UK) and a matched control sample from a UK based Community Mental Health Team (CMHT) over a 12 month period. IT Malta included HTRYP 13-25 years and IT UK 15-25 years. An in-depth multidisciplinary team (MDT) assessment was carried out together with the Health of the Nation Outcome Scales for Child and Adolescent Mental Health (HoNOSCA) and Children’s Global Assessment Scale (CGAS) at assessment and discharge. Those meeting criteria were offered intensive individualised tailored therapy.
Results: 38 YP were referred to IT Malta, 34 (89.5%) assessed and 22 (57.9%) treated in 2018. 40 referrals to IT UK, 36 (90%) assessed, 31 (86%) met criteria and 15 (48%) were treated. The HoNOSCA for IT Malta and IT UK were similar, at baseline (20.65, 19.32) and discharge (16.65, 16.53) respectively, but higher when compared to CMHT (11.19, 8.03) (p <0.001). The mean baseline CGAS for IT Malta (46.0) elucidates more impairment compared to IT UK (51.1) and CMHT (58.9). The total mean contact times (hours) per YP were; IT Malta (24.5), IT UK (18.2) and CMHT (8.6).
Conclusion: This study identified that in both the Maltese and UK population there exist a cohort of HTRYP who are not accessing services and are not addressing their mental health needs. Both IT Malta and IT UK identified HTRYP and through intensive outreach work, were able to engaged and treat these. HTRYP may benefit from a flexible YP oriented service, including outreach capacity and a focus on engagement. Despite improvement,
Keywords
Adolescence, Assessment, Complex Mental Disorder, Hard to Reach Young People, Innovative, Intervention, Mental Health, Prognosis
Article Details
Abbreviations:
CGAS: Children’s Global Assess-ment Scale; CMHT: Community Mental Health Team; CYPS: Children and Young People’s Service; HRTYP: Hard-to-Reach Young People; HoNOSCA: Health of the Nation Outcome Scales for Child and Adolescent Mental Health; IT: Innovations Team; MDT: Multidisciplinary Team; UK: United Kingdom; YP: Young People
1. Background
As part of normal development, adolescents negotiate multiple transitions in many aspects of their life, these include furthering education or employment and becoming independent so as to individualise into unique adults. Often the adolescent period has been described as one of high-risk. Many young people (YP) tend to be “unprepared for transitions” or ill-equipped for the multiple challenges of adolescence and consequently, their mental disorders may suffer (p <0.05) [1]. These transitions compounded with other psychosocial stressors make YP more vulnerable to particular risks such as mental disorders [1]. 5-10% of YP are defined as ‘hard to reach’ (HTRYP) [2]. These HTRYP are particularly vulnerable individuals who are at risk of coming from disadvantaged back-grounds, ending up being marginalised, sometimes homeless and who often slip through the health care system or are unwilling to engage in services [1]. For the purpose of this study the term HTRYP will be used to describe such a person.
To access specialised services, a referral by a GP is required, which to a YP with complex needs may be a barrier to access services [3]. Furthermore, most mental health services are associated with high levels of stigma. Another barrier to accessing mental health services is age [4]. Singh et al. [5] in a systematic review on transition of YP between children to adult services, report that 30-60% of YP slip through the ‘service gap’ and as a result are lost to follow up [6]. Gonzalez [7] emphasised that when children are engaged in a project it increases the validity of the findings. The Innovations Team (IT) UK 15 to 25 years was a multidisciplinary team (MDT) based in the North East of England throughout 2011 and was developed to provide a service which identified, assessed and treated HTRYP [1]. The IT Malta is a specialist community-based service established in January 2018 within the National Children and Young People’s Service (CYPS), Malta for YP aged 13 to 25 years who could not be managed within existing services. The aim for this team was to offer an intensive and flexible service, which focused on using an in-depth multimodal developmental assessment, followed by individualised community-based outreach care plans. The flexibility was provided through weekly MDT meetings, changes to the care plan according to the unique needs of the individual, offering home visits if the YP was too unwell to attend their appointment, facilitate engagement with the service and their local community. Attendance rates were improved by regular calls to patients and/or family to confirm appointments and follow up even if they failed to attend reviews.
The aim of this study was to compare the service offered, demographics, mental disorders, and social functioning of the IT Malta to a similar UK service (IT UK) and with a matched sample from a community mental health team (CMHT). The latter; was a CMHT also based in the North East England. The alternative hypothesis was that there were no significant clinical differences found between IT Malta and IT UK, however a statistically significant change would be found in the mental state and social functioning of IT Malta and IT UK from TP1 (baseline) to TP2 (discharge) when compared to CMHT cohort of YP over a maximum period of twelve months.
2. Methods
This case control study compared the outcomes from
two service evaluations (IT Malta and IT UK) and a matched group from a UK based CMHT. The organ-isational setup of the IT Malta service was mapped onto IT UK, meaning the team consisted of a MDT having different child and adolescent mental health training backgrounds. The same protocol used in IT UK was amended and adapted for use in IT Malta [1]. In Malta, the age group was extended to include 13- and 14-year-olds seeing as referrals were made for younger patients. In the UK, a psychology assistant would call to remind patients of their appointments and to help to improve engagement whereas in Malta this was done conjointly by a psychology assistant and nurse. The IT Malta and IT UK inclusion criteria were very specific to ensure the selection of only YP who presented with multiple complex mental disorders according to ICD-10 [8] and who presented with a complex social background which negatively affected their function within society. Furthermore, these YP had to fulfil the criteria as being hard-to-reach, which meant that either they had been in contact with mental health services but were dissatisfied (74%) or else they had never been in contact but still were unwilling to access mental health services and thus had unmet mental health needs [9]. Only those YP who met the criteria as defined in the study protocol as HRTYP were included in this study and this was decided following an assessment procedure as described below and MDT meeting.
To decrease barriers to access services, both IT Malta and IT UK accepted referrals from: GPs, social services, schools, self-referrals or walk-ins. Both IT services carried out a multimodal in-depth clinical assessment which consisted of a clinical psychiatric interview carried out by a child and adolescent psych-iatrist. This entailed a neurodevelopmental assessment which was substantiated using the Mini-International Neuropsychiatric Interview for Children and Adol-escents (MINI Kid) [10]. A social function interview was carried out by mental health nurses or a social worker, this was substantiated by the Assess-mentSchedule for Adolescents (SNASA) [11]. Psychologists carried out a systemic clinical interview informed using Systemic Clinical Outcome and Routine Evaluation (SCORE-15) [12].
Following the completion of the described assessment every YP would be discussed at an MDT formulation meeting. Only those who met the criteria described within the service protocol for HTRYP [1] and deemed to be suffering from complex mental disorders would be taken on for treatment. Those YP who did not meet criteria would be discharged from IT UK or IT Malta and referred to other appropriate services. HTRYP taken on for intervention would be allocated a key worker who would be responsible for coordin-ating the intervention plan and ensuring appropriate liaison with external agencies. All interventions would be tailored to meet the needs of the YP (therefore were not manualised). Initially the mainstay of intervention would focus on engagement. This was followed by providing the appropriate time needed (at times more than once weekly) to apply evidence-based inter-ventions. This consisted of psychopharmacology, home visits, outreach meetings held in the community, supportive psychotherapy, cognitive behaviour thera-py, dialectical behaviour therapy skills (DBT), family therapy, social worker involvement and if required admissions to in-patient units. The outcome measures consisted of the Health of the Nation Outcome Scales for Child and Adolescent Mental Health (HoNOSCA) and Children’s Global Assessment Scale (CGAS). HoNOSCA scores are reported [13] to have good inter-rater reliability (0.82 for psychiatric symptoms and 0.42-0.62 for physical and social impairment), good test re-test reliability 0.69 (p<0.001, two tailed Pearson correlation), interclass correlations greater than 0.8 and good face validity [14]. The CGAS is widely available and reported to have good joint reliability of 0.83-0.92, good inter-rater reliability and a useful measure of change over time [15].
Data from IT Malta were compared with IT UK and CMHT using SPSS [16]. YP from IT UK and CMHT were matched for age, gender, education and socio-economic status. In the initial phase of the study, all YP attending CMHT between January to December 2011 were identified through patient electronic data-base. It was decided that only those YP matched on age and date of discharge with the IT UK and IT Malta would be included in this study. The rationale behind this was to have YP of a similar age and requiring intervention for a similar period, not extending beyond 1 year were included. At the end of the year, only those YP from CMHT which had similar age, gender, education, and socioeconomic status were included for data collection for service input and outcome meas-ures. The reasoning was to include only YP with similar sociodemographic variables, as it was believed that including higher functioning YP would falsely skew the outcome data, either showing an overly posit-ive outcome from CMHT or on the other hand, a much more significant improvement of the IT compared to the CMHT [1]. The Chi Squared test was used to investigate the association between two categorical variables whereas the one-way ANOVA test was used to compare mean quantitative measurements between the 3 independent groups: CMHT (UK control), IT UK (UK case), IT Malta (Malta case). The Tukey Post-HOC test was essential to compare the mean measurements between the groups two at a time.
3. Results
38 YP referred were accepted by IT Malta for assessment, 34 (89.5%) were assessed, 11 (28.9%) were discharged, 5 (13.1%) refused assessment and 22 (57.9%) received intensive treatment by the end of 2018. Comparatively for IT UK, 40 referrals were received, 36 (90%) were assessed over the one-year period, 31 (86%) met criteria for HTRYP and 15 (48%) were offered individually tailored therapy. Both services were compared to 115 systematically selected and matched YP who attended CMHT. The mean age at referral was 16.21 years (SD 2.8) for IT Malta, which was lower than IT UK (mean 18.48, SD 2.3) and CMHT (mean 19.34, SD 2.9) p < 0.001. Most referrals to CMHT were received from GPs (71.4%) whereas CYPS (24.4%), looked after children (26.8%) and GPs (17.1%) mainly referred to the IT UK. There were no direct referrals from GPs to IT Malta and YP were mainly referred from CYPS (33.3%), walk-in (25%), adult services (20.8%) or directly from the in-patient psychiatry hospital upon discharge (16.7%), X2(20) = 109.38, p < 0.001. The majority of YP referred to both IT services had previous service involvement when compared to CMHT (X2(2) = 23.596, p < 0.001).
IT Malta had a significantly longer waiting time for the initial assessment when compared to both UK services (p= 0.009), Table 2. The number of missed or cancelled appointments of the HTRYP and the overall attendance rate were similar for both the IT Malta and IT UK services although this was not statistically significant in comparison to CMHT (p= 0.41). The total mean contact time per YP were IT Malta (24.5 hours), IT UK (18.2 hours) and CMHT (8.6 hours). The HoNOSCA scores at baseline and discharge for both the IT Malta and IT UK services were similar. Both IT services had higher HoNOSCA scores which inferred a greater severity of mental disorder when compared with the CMHT group (p <0.001), Figure 1. The CGAS at baseline for IT Malta (mean 46.0, SD ±9.4) elucidates more impairment when compared to IT UK (mean 51.1, SD ±14.0) and CMHT (mean 58.9, SD ±5.1). Using Tukey Post-Hoc Test there was a significant difference between baseline CGAS of IT Malta and CMHT (p = 0.006, SE 4.1) however there was no significant difference found between the three groups for CGAS on discharge (p = 0.18), Figure 2.
The mean number of diagnosis per YP were similar for both the IT Malta and IT UK but differed significantly from the CMHT X2(36) = 95.89, p < 0.001. The most common diagnosis in the CMHT was depression (56.3%) and anxiety disorder (23.9%), whereas in IT UK, 52.5% were diagnosed with substance misuse, 37.5% had anxiety and depression and 32.5% had antisocial personality disorder (ASPD), conduct disorder (CD) or oppositional defiant disorder (ODD). The commonest disorder diagnosed in IT Malta was anxiety (30.4%), followed by attention deficit hyper-activity disorder and attachment disorder (26.1%) and post-traumatic stress disorder and ASPD/CD/ODD (21.7%). The most prescribed medication for IT UK was Atomoxetine (60.0%) whereas only 18.3% YP in CMHT and 0% in IT Malta where prescribed this medication. Hypnotics were more commonly prescri-bed in the CMHT (36.6%) when compared to both the IT UK (8.6%) and IT Malta (0%). The most common medication prescribed in IT Malta were selective serotonin reuptake inhibitors (SSRI) (50%), Table 4. The main modality of therapy for the IT Malta was supportive, motivational and DBT (58%) whereas home-based intervention in the community was the main focus for IT UK (20%) and CMHT (14.1%) as shown in Table 5.
UK (Control) |
UK (Case) |
Malta (Case) |
p value |
||
Gender |
Female |
70.60% |
61.30% |
31.60% |
X2(2) = 15.41, p < 0.001 |
Male |
29.40% |
38.70% |
68.40% |
||
Accommodation at referral |
Stable |
96.90% |
51.60% |
85.30% |
X2(4) = 32.216, p < 0.001 |
Unstable |
3.10% |
41.90% |
14.70% |
||
Homeless |
0.00% |
6.50% |
0.00% |
||
In education/ employment |
Employed |
3.10% |
0.00% |
18.80% |
X2(6) = 76.69, p < 0.001 |
Unemployed |
0.00% |
71.00% |
43.80% |
||
Long sick leave |
0.00% |
3.20% |
0.00% |
Table 1: Comparison of demographics between IT Malta, IT UK and CMHT (UK).
Mean |
Std. Deviation |
P-value |
||
Days Awaiting Initial Assessment |
CMHT (Control) |
12.18 |
12.105 |
0.009 |
IT UK (Case) |
18.68 |
21.985 |
||
IT Malta (Case) |
30.35 |
45.773 |
||
Number of cancelled appointments |
CMHT (Control) |
2.61 |
3.223 |
0.405 |
IT UK (Case) |
3.22 |
3.125 |
||
IT Malta (Case) |
3.54 |
3.956 |
||
Total Contact time minutes |
CMHT (Control) |
518.59 |
910.235 |
0.101 |
IT UK (Case) |
1090.11 |
1640.86 |
||
IT Malta (Case) |
1470.63 |
4166.265 |
||
Attendance rate % |
CMHT (Control) |
69.1 |
30.207 |
0.021 |
IT UK (Case) |
54.04 |
29.644 |
||
IT Malta (Case) |
55.46 |
33.606 |
Table 2: Comparison service input between IT Malta, IT UK and CMHT (UK).
Baseline 95% CI: UK Control [9.89, 12.50], UK Case [16.72, 21.93], Malta Case [18.05, 23.25]. Discharge 95% CI: UK Control [6.52, 9.53], UK Case [12.05, 21.01], Malta Case [13.13, 20.16].
Baseline 95% CI: UK Control [52.95, 64.83], UK Case [46.56, 55.64], Malta Case [41.53, 50.37]. Discharge 95% CI: UK Control [46.26, 69.45], UK Case [50.70, 67.42], Malta Case [44.13, 56.46].
Treatment |
CMHT (Control) |
IT UK (Case) |
IT Malta (Case) |
SSRI |
29 (40.8%) |
5 (14.3%) |
6 (50.0%) |
Methylphenidate |
5 (7.0%) |
1 (2.9%) |
2 (16.7%) |
Atypical antipsychotics |
1 (1.4%) |
1 (2.9%) |
2 (16.7%) |
Mood stabilizers |
11 (15.5%) |
2 (5.7%) |
2 (16.7%) |
Hypnotics |
26 (36.6%) |
3 (8.6%) |
0 (0%) |
Atomoxetine |
13 (18.3%) |
21 (60.0%) |
0 (0%) |
Table 3: Medication prescribed by each service X2(30) = 111.20, p < 0.001.
Therapies |
CMHT (Control) |
IT UK (Case) |
IT Malta (Case) |
CBT |
8 (11.3%) |
0 (0%) |
2 (16.7%) |
Supportive psychotherapy |
3 (4.2%) |
4 (11.4%) |
7 (58.3%) |
Motivational therapy/ DBT, other therapies |
3 (4.2%) |
1 (2.9%) |
7 (58.3%) |
Home treatment |
10 (14.1%) |
7 (20.0%) |
2 (16.7%) |
Table 4: Therapies offered X2(30) = 111.20, p < 0.001.
4. Discussion and Conclusions
Both the IT Malta and IT UK were able to identify, engage and treat YP with multiple complex mental disorders whose needs were not being met by other mental health services. There was no statistical difference between the baseline HoNOSCA and CGAS for both IT Malta and IT UK, indicating that both services were able to identify YP who although not registered within mental health services, were living in the community and suffering from a greater degree of severity of mental disorder than those YP attending the CMHT. Both innovations services had statistically and clinically worse baseline scores than the CMHT. Both IT Malta and IT UK identified YP who were suffering from more than one mental disorder which is greater than was reported in the literature [17]. The HTRYP identified were suffering from complex mental disorders to the extent that this was affecting their social functioning as evidenced by the demographics, worse HoNOSCA and CGAS scores, higher number of diagnoses and more treat-ments prescribed. This finding holds true for both countries regardless of the very different cultural backgrounds, clinical settings and clinicians; all of these acting as confounding variables.
This study contributes towards the literature through the fact that similar findings were found for: the presence of HTRYP with similar severity of mental disorder and, in two different clinical settings, one an inner-city area in the UK and the other, a Mediter-ranean island. This provides evidence to substantiate the generalisability of the findings of this study and thus HTRYP are not solely unique to the inner cities of the UK but are possibly present in every country. Given that YP in IT UK and IT Malta were able to attend for an in-depth assessment and did engage with respective services is evidence that provides hope for the clinician working with YP. This paper would like to emphasis the fact that although HTRYP initially present as challenging when it comes to engagement and more severely unwell, from the findings of both the IT Malta and IT UK it seems that the majority of HTRYP do engage with clinicians and through presence of an outreach team offering time to facilitate the engagement process and to accurately pace the intensive therapy with a high staff to low patient ratio, regular MDT and unique and individual care plans discussed and revisited every week. As shown in Camilleri (2017) [1] it also seems that the term HTRYP defines a YP at a particular moment in time rather than being a ‘negative’ permanent label which defines a YP in our communities.
The lower mean age in Malta could be the result of accepting younger (ages 13 and 14 vs IT UK age 15) YP into the service. It could also be that since IT Malta was based within the CYPS, the HTRYP were identified early from CYPS and referred to IT Malta. The gender difference may be explained as depression was more commonly diagnosed in IT UK and CMHT when compared to Malta and post-puberty depression would be more prevalent in females [18]. IT Malta dedicated triple and IT UK double the amount of time per YP when compared to the CMHT. This study postulates the importance of ‘time’ being a key factor to successful engagement with a HTRYP and offering a clinically effective treatment. A dedicated outreach team with a high staff to patient ratio working specifically on engaging YP with complex mental disorders and offering them an individualised tailored management plan, was clinically effective.
The significant proportion (frequency 22.1%) of HTRYP with neurodevelopmental disorders supports the concept that a developmental approach is relevant and should be used in assessments by psychiatric services into early adult life [1]. A number of HTRYP may have had previously unsatisfactory contact with mental health services [1]. However, through the availability of a more flexible service which facilitated access of referrals into the service, provided a focus on engagement through the use of home and community reviews, the variety of mental health professionals with different training backgrounds, the repeated attempts at contacting the YP via a nurse or psych-ology assistant rather than receiving cold calls through a letter or clerk and having a MDT with a different set of psychotherapeutic skills increased the possibility of offering a clinically effective intervention to these HTRYP. This intervention helped to reduce their morbidity and mortality through mental disorders [19].
The longer waiting times recorded by IT Malta could be a result of the stigma associated with the location (CYPS) of this service opposed to being located within a GP surgery like IT UK or, perhaps because IT Malta staff, although were better staffed in terms of numbers than IT UK, were not working within this service full time. Therefore, a recommendation from this study is for such future services to be based outside of mental health facilities and within the community or GP surgeries. Generalisability of these findings needs to be done with care since to date no Maltese control group has been identified and compared. However, both IT UK and CMHT groups were matched on several sociodemographic factors. Another limitation of this study was the small sample sizes of both IT services. This was expected considering the nature of the characteristics of the targeted cohort. No formal reliability testing for the data collection process was undertaken, although to minimise any bias a trained clinical researcher carried out the double data checking.
The next steps would be to carry out a qualitative and quantitative follow up study and compare the results of both these services with a control CMHT in Malta. This would help elucidate whether the positive clinical change following intervention had a lasting effect on the trajectory of the lives of these YP. Although there was an improvement in the mental state and social function of both IT Malta and IT UK over time, the mental state and social function of these HTRYP remained clinically worse than those of the control group. A recommended study would be to identify moderators and mediators that could help identify which HTRYP suffering from which disorders would benefit from which specific psychotherapeutic inter-ventions.
Declarations
Ethics approval and consent to participate
NHS UK ethical approval was gained in June 2013 (ref: 13/NE/0150) with amendment approved in April 2014. Caldecott approval together with research and developmental approval in the relevant NHS organi-sations and Mental Health Services Malta were also gained.
Consent for publication
Andrea Saliba and Nigel Camilleri give their consent for this study to be published in the Journal of Pediatrics, Perinatology and Child Health.
Availability of data and materials
Authors confirm that they have full access to all the data in the study and take responsibility for the integrity of the data in the study and the accuracy of the data analysis.
Competing interests
The authors have declared that they have no com-peting or potential conflicts of interest. The Innova-tions Project was funded by the North East Strategic Health Authority. The study was funded by the Malta Government Scholarship Scheme, and was a National Institute for Health Research (NIHR) Portfolio Study.
Funding
The authors are not currently in receipt of any research funding for this study.
Authors' contributions
Both authors have made substantial contributions to the conception and design of the study and have approved the submitted version. Both authors agree to be personally accountable for their own contributions and for ensuring that questions related to the accuracy or integrity of any part of the work, even ones in which the author was not personally involved, are appro-priately investigated, resolved, and documented in the literature.
Acknowledgements
References
- Camilleri N, Newbury-Birch D, McArdle P, et al. Innovations in Practice: A case control and follow-up study of ‘hard to reach’ young people who suffered from multiple complex mental disorders. Child and Adolescent Mental Health 22 (2017): 49-57.
- Social Exclusion Unit. Breaking the cycle – Taking stock of progress and priori-ties for the future. London: Office of the Deputy Prime Minister (2004).
- Patel V, Flisher AJ, Hetrick S, et al. Mental health of young people: a global public-health challenge. The Lancet 369 (2007): 1302-1313.
- Rickwood DJ, Deane FP, Wilson CJ. When and how do young people seek pro-fessional help for mental health problems?. The Medical journal of Australia (2007).
- Singh SP, Paul M, Ford T, et al. Process, out-come and experience of transition from child to adult mental healthcare: Mul-tiperspective study. British Journal of Psychiatry 197 (2010): 305-312.
- Gondek D, Edbrooke-Childs J, Velikonja T, et al. Facilitators and Barriers to Person-centred Care in Child and Young People Mental Health Services: A Systematic Review. Clinical Psychology and Psycho-therapy 24 (2017): 870-886.
- Gonzalez M, Phoenix M, Saxena S, et al. Strategies used to engage hard-to-reach populations in childhood disability re-search: a scoping review (2020).
- Organization WH. Global health risks: mortality and burden of disease attributa-ble to selected major risks (2009).
- Camilleri N, Newbury-Birch D, McArdle P, et al. Innovations in Practice: A case control and follow-up study of ‘hard to reach’ young people who suffered from multiple complex mental disorders. Child and Adolescent Mental Health 22 (2021): 49-57.
- Sheehan D V, Sheehan KH, Shytle RD, et al. Reliability and validity of the mini international neuropsychiatric interview for children and adolescents (MINI-KID). Journal of Clinical Psychiatry 71 (2010): 313-326.
- Kroll L, Woodham A, Rothwell J, et al. Relia-bility of the Salford Needs Assessment Schedule for Adolescents. Psychological Medicine 29 (1999): 891-902.
- Stratton P, Bland J, Janes E, et al. Developing an indicator of family function and a practicable outcome measure for systemic family and couple therapy: The SCORE. Journal of Family Therapy 32 (2010): 232-258.
- Gowers SG, Harrington RC, Whitton A, et al. Brief scale for measuring the outcomes of emotional and behavioural disorders in children Health of the Nation Outcome Scales for Children and Adolescents (HoNOSCA) (1999).
- Garralda M, Yates P, Higginson I. Child and adolescent mental health ser-vice use: HoNOSCA as an outcome measure. The British Journal of Psychiatry (2000).
- Green B, Shirk S, Hanze D, et al. The Children’s Global Assessment Scale in Clinical Practice: An Empirical Evaluation. Journal of the American Academy of Child & Adolescent Psychiatry 33 (1994): 1158-1164.
- Green SB, Salkind NJ. Using SPSS for Windows and Macintosh (7th Edition). Pearson (2013): 448.
- Jenkins R, Meltzer H, Bebbington P, et al. The British Mental Health Survey Programme: Achievements and latest findings. So-cial Psychiatry and Psychiatric Epidemiology 44 (2009): 899-904.
- Cohen P, Cohen J, Kasen S, et al. An Epide-miological Study of Disorders in Late Childhood and Adolescence?I. Age- and Gender-Specific Prevalence. Journal of Child Psychology and Psychiatry 34 (1993): 851-867.
- McGorry PD, Purcell R, Hickie IB, et al. Investing in youth mental health is a best buy. Medical Journal of Australia 187 (2007): 5-7.