Hong Kong J Psychiatry 2007;17:124-30

ORIGINAL ARTICLE

Dr Roger MK Ng, MBChB, MSc (Birm), MRCPsych (UK), FHKAM (Psychiatry), FHKCPsych, Department of Psychiatry, Kowloon Hospital, Hong Kong, China.
Dr Martina SM Cheung, MSocSc (Clinical Psychology), Department of Psychiatry, Castle Peak Hospital, Hong Kong, China.

Address for correspondence: Dr Roger MK Ng, Department of Psychiatry, 7/F, Main Block, 147A, Argyle Street, Hong Kong, China. Tel: (852) 3129 7111; Fax: (852) 2711 4601;
E-mail: ngmk@ha.org.hk

Submitted: 16 July 2007; Accepted: 27 August 2007

pdf Full Paper in PDF

Introduction

Cognitive behavioural therapy (CBT) for psychosis is now a recommended adjunct treatment for treatment-resistant psychotic patients in the United Kingdom’s NICE (National Institute for Clinical Excellence) guidelines.¹ There is ample evidence to support the efficacy of CBT for the reduction of positive symptoms in medication-resistant psychosis.² There are case reports and uncontrolled trials in Asia that also support CBT for psychosis.³ Tarrier⁴ has highlighted some of the difficulties encountered with the dissemination of CBT in a daily clinical setting. One of the most prominent obstacles is that there is a lack of well-structured training programmes for novice therapists. Yet there is a dearth of data about what constitutes an effective training programme. The exact contribution of didactic and experiential training toward effective training in CBT for psychosis is not clear. If didactic learning is sufficient for acquiring essential skills in CBT for psychosis, it will potentially reduce the amount of training time needed. Teasing out the relative contribution of didactic and experiential training in a training programme is therefore of practical importance.

Cognitive behavioural therapy supervision can be as complex and challenging as CBT itself. Thus it is surprising that so little has been written about CBT supervision.⁵ A recent systematic review⁶ and a local study in Hong Kong⁷ have suggested that CBT supervision may lead to increased therapist competence and subsequent clinical improvement in patient symptom severity. Yet there are limited data about the impact of supervision on therapist competence in CBT for psychosis. Supervised training can be viewed as improving 2 different types of knowledge in the trainees, namely declarative knowledge and procedural knowledge.⁸ Declarative knowledge refers to factual knowledge about the theories and practice of CBT (for example, current evidence of the efficacy of CBT for psychosis on first-episode psychotic patients). Declarative knowledge is typically learnt didactically through lectures and reading assignments. Such inert knowledge can only be transferred into practical skills through enactive training like role-plays, actual clinical practice and supervision. Procedural knowledge refers to the rules, plans, and procedures that lead to the direct application of skills. There is indirect evidence suggesting that imparting good declarative knowledge without transfer to procedural knowledge will affect the quality of therapy supervision. It has been found that ‘traditional supervision’ characterised by didactic teaching and direct feedback on therapy sessions is inferior to systematic training characterised by specific interpersonal skills training and actual implementation of skills in supervision.⁹ Nonetheless, this important issue is far from being settled. The current study aimed to investigate the effect of a CBT training programme on the change in declarative and procedural knowledge in a group of mental health care workers in a non-governmental organisation (NGO) in Hong Kong.

Methods

Sample

Trainee Therapists

The sample was composed of 12 experienced care professionals who were working in a Hong Kong NGO providing a range of rehabilitation services for people suffering from serious psychiatric disorders (halfway houses, sheltered workshops, supported employment services and community outreach). They were recruited via recommendations from senior managers who selected the trainees based on their expressed interest in learning CBT and the nature of the trainees’ work. Therefore, a trainee was only nominated for training if they expressed definite interest in learning CBT for psychosis and if they were taking care of psychotic patients on a regular basis.

Patients for Cognitive Behavioural Therapy

Each trainee was required to take up 1 patient suffering from medication-resistant psychosis for CBT for a period of not less than 6 months. The patients were all halfway house residents under the care of this NGO, suggesting that the subjects were psychotic patients with rather low levels of independent living skills. The selection criteria included: (1) patients taking high doses of medication (defined as more than 600 mg of chlorpromazine equivalents) but were still experiencing persistent positive symptoms in the form of delusions and hallucinations; (2) able to give written consent to treatment; (3) able to speak and understand Chinese. Exclusion criteria included the presence of a co- morbidity like post-psychotic depression or substance abuse, and patients who were actively suicidal.

The Trainer

The trainer (first author) is a 40-year-old male Chinese psychiatrist with over 16 years of clinical experience with psychotic patients. He is also a certified cognitive therapist and cognitive therapy trainer / supervisor. He has over 10 years of experience with CBT for psychosis. He has over 5 years of experience of supervising psychiatric trainees in the application of CBT for a wide range of psychiatric problems. He has also published papers on CBT for psychosis and CBT supervision. This was the first group supervision and training course for social workers on CBT for psychosis organised by the trainer.

The Training Programme

The training programme was intended to prepare trainee therapists for a pilot study on the effectiveness of CBT for positive symptoms in medication-resistant psychotic patients. The training programme consisted of 2 phases: the first phase (didactic lecture phase) was a once weekly, 12- week training programme that lasted for a total of 25 hours, and the second phase (group supervision training phase) was a weekly supervision group training wherein the supervisees discussed the patients they were treating with CBT. The topics covered in the didactic phase included: (1) basic techniques for CBT; (2) cognitive formulation of delusions; (3) cognitive model of voices and negative symptoms; (3) cognitive and behavioural techniques for delusions; (4) cognitive and behavioural techniques for voices and negative symptoms; (5) schema work; (6) cognitive and behavioural interventions for medication adherence problems; (7) relapse prevention work; (8) case formulation for psychosis and its relationship with intervention strategies; (9) relationship ruptures and repair of the therapeutic alliance; (10) current evidence for CBT for psychosis and a brief update on other new interventions like mindfulness and acceptance and commitment therapy. Trainees were also assigned homework reading after each didactic lecture, based on the therapists’ manual ‘cognitive therapy for psychosis’.¹⁰ Specific time was allocated at the beginning of every lecture for feedback on the reading assignments.

The second phase consisted of weekly group supervision by the trainer (author) for the supervisees. Due to the supervisees’ work commitments, only 6 to 8 could attend each supervisory session. Each supervisory session was composed of 2 parts: discussion of the progress of 2 or 3 patients and a review of a pre-determined, supervisee’s therapy sessions with the supervisees. Each trainee was required to submit every videotaped session for recording and research purposes. The skills used in cognitive therapy like setting up of behavioural experiments and completion of dysfunctional thought records were discussed in both the didactic sessions and in the group supervision. In the group supervision, such skills were focused on and coached using role-plays and role-reversals. Cognitive therapy skills (what, how, and when to apply certain skills) were applied with explicit referral to the case formulation of each particular patient. The review of a pre-determined videotaped session with trainees provided a particularly important experiential exercise, as trainees were encouraged to give constructive comments on the performance of the trainee in that particular session and to role-play a problematic segment. Such role- plays would then be followed up by extensive discussion of the problems experienced by the patient and the therapist. Therapist skill issues in the problematic video segment were examined using the declarative-procedural-reflective (DPR) model.¹¹ In the DPR model, therapist skills can be examined in 3 areas (reflective skills, declarative knowledge, and procedural systems). Problems in these 3 areas were identified and then targeted for improvement during group supervision sessions. Declarative knowledge (technical skills or therapeutic alliance) would usually be dealt with by explicit referral to Kingdon & Turkington’s manual¹² and reading assignments on up-to-date research papers. Procedural skill deficits were targeted through identification of specific problems (interpersonal perceptual skill deficits, therapist attitudes, or interpersonal active communication skill deficits). As suggested by Bennett-Levy,¹¹ the training of novice therapists places heavy emphasis on the declarative and procedural systems, while developing expertise requires extensive self-reflection. In this training module for novice therapists, active self-reflection is not an explicit focus of training.

Outcome Measurements

Declarative Knowledge

Declarative knowledge about CBT for psychosis was assessed by testing knowledge about preparing cognitive- behavioural formulation for 2 case vignettes. To cognitive therapists, case formulation is the cornerstone of evidence- based practice.¹³ Skill in case formulation has been found to be related to the common factors of psychotherapy, which Wampold et al¹⁴ estimated, collectively explained about 70% of the total variance in therapy outcomes. The functions of this conceptual scheme are to instill hope in the patient, provide new learning experiences, arouse emotions, enhance self-efficacy, and provide opportunities to practise.¹⁵ Case formulation can also enhance the therapeutic alliance, increase therapist empathy for the client through understanding gained from the formulation, and increase therapist confidence.¹⁶

In a review of the literature on the case formulation system, Bieling and Kuyken¹⁷ concluded that scientific case formulation must fulfil the following criteria: (1) reliability; (2) quality; (3) treatment utility; and (4) applicability. Kuyken et al¹⁸ argued that replicable case formulations do not imply high quality ones and a high intervention impact. They found that the quality of case formulation was directly related to the level of experience and accreditation status of professional cognitive therapy organisations.¹⁸ To assess the quality of case formulations produced by the trainees in this study, a scale called the Quality of Cognitive Case Formulation Rating Scale (QCCFRS) was used. This rating scale was developed as a way of establishing the quality of cognitive therapy case formulations in a study of 115 mental health professionals and their case formulation drafting after training workshops.¹⁸ Quality is defined as a parsimonious, coherent, and meaningful account of a client’s presenting problems in cognitive therapy terms. This refers to whether the formulation is coherently structured, with the elements within the formulation interacting with the presenting problems in a meaningful manner across situations or time. This is premised on the assumption that a parsimonious, coherent, and meaningful diagram is the basis for focused and high impact interventions. ‘Best practice’ in high-quality cognitive therapy training programmes was used to operationalise the scale. Initial phases involved establishing what made up a ‘good enough’ case formulation. This process led to the scaling of the quality of case formulations from ‘good’, to ‘good enough’, to ‘poor’ and to ‘very poor’. A ‘good enough’ formulation includes most relevant information (childhood data, core beliefs, dysfunctional assumptions, and compensatory strategies) in appropriate detail, and links this information to prototypical problematic situations. A ‘very poor’ formulation shows minimal integration of the elements, includes much irrelevant information and shows evidence that the therapist misunderstands the case formulation.

Two case vignettes used in the current study were adapted from a book.¹⁹ One case vignette, which focused on the formulation of persecutory delusions in paranoid schizophrenia, was adapted from a chapter²⁰ of the case book. The other vignette, on the formulation of persistent auditory hallucinations, was obtained from another chapter.²¹ Both case vignettes were modified with demographic features and living arrangements adapted for Chinese culture. The trainees were required to read the case vignettes and write up a cognitive behavioural formulation for each vignette within a time frame of 1 hour under the direct observation of the trainer. The trainees were also strongly advised not to discuss the case vignettes among themselves during the assessment. The trainer collected the assessments after completion and no model answers were provided thereafter. This first assessment was conducted after the completion of the 10-week didactic lectures, yielding a total of 24 case formulations (done by 12 trainees). The trainer did not inform the trainees about the re-administration of the case vignettes after the completion of the 6-month group supervision. The trainees were then invited to re-complete the case formulations for the same 2 case vignettes at the end of the group supervision. At this assessment, 10 trainees produced 20 case formulations: 2 trainees had dropped out.

The first author rated all the case formulations using the QCCFRS at the end of the entire training period. The case formulations were also sent to the second author for rating, using the same scale, after completion of the training period. The second author was blinded to the identity of the trainees and the dates of completion of the case formulations. The second author was unaware that she was rating 44 case formulations produced by the trainees at 2 different times. Before rating the case formulations, the 2 authors discussed the rating method based on the QCCFRS. The inter-rater reliability between the first and second authors was calculated using a kappa agreement measure, and attained a value of 0.85, which is a high level of agreement, on a separate group of case formulations. Both raters are experienced cognitive therapists with extensive experience in CBT for psychosis. Both are also certified cognitive therapists of the Academy of Cognitive Therapy (United States), suggesting that they have attained a certain level of expertise in the delivery of cognitive therapy for an extensive range of psychiatric patients.

Procedural Skills

Procedural skills used for the actual conduct of cognitive therapy were assessed by rating the third sessions and the second session before the final one with the Cognitive Therapy Rating Scale (CTRS).²² The Scale is an 11-item scale that measures 2 therapy skill areas: general therapeutic skills (items 1-6) and specific technical cognitive therapy skills (items 7-11). For each item, the rating score ranges from 0 to 6. The Scale is a psychometrically validated scale for measuring cognitive therapists’ competence. It has been found to have high inter-rater reliability (the reliability of a single rater is 0.59, which is moderate for a psychotherapy rating scale), and strong discriminant validity for distinguishing acceptable and unacceptable cognitive therapy sessions.²³ A generally supportive therapy session without cognitive therapy skills is defined as one in which each item gets a score of 2 out of 6 and a total score of 22. An effective cognitive therapy session is defined as one in which each item reaches a score of at least 4 out of 6 and a total score of not less than 44.²⁴

The rationale for excluding the first 2 and the last sessions is that the first 2 sessions usually focus on history taking while the last session emphasises making a conclusion and an agreement for future contacts. Hence the sessions in between were the most suitable for assessing a range of cognitive behavioural skills. Using the third session and the penultimate one as those to be rated with CTRS would best provide data for testing the hypothesis that there would be a change in procedural skills after a period of group supervision. The author performed the CTRS rating for each videotaped session. The author had previously attained a high level of inter-rater reliability level with another expert cognitive therapist from the Beck Institute when using CTRS in another supervision study.⁷

Statistical Analysis

The SPSS package 11.0 was used for statistical analysis. All parametric data were summarised as means with standard deviations. Non-parametric data (the CTRS rating) were normalised by using a logarithmic conversion and then subjected to a paired t test. The statistical significance level was pre-set at a p value of less than 0.05. For categorical data (the quality of case formulation), the proportion of trainees who were ‘satisfactory’ (defined as ‘good enough’ or ‘good’) or ‘unsatisfactory’ (defined as ‘very poor’ or ‘poor’) was compared using the Chi-square test. Again the statistical significance was set at a p value of less than 0.05.

Results

Trainees’ Background

All trainees were working in the same NGO, one that provides residential and employment services for patients with psychiatric problems. At the time of recruitment, 12 workers were nominated by the director and accepted for training. There were 4 Chinese males and 8 Chinese females. All were registered social workers except 1 female trainee who was an enrolled nurse by profession. The mean age of the participants was 38.7 years (range, 28-46 years). Three had a Master’s Degree in Social Work, 4 had a Bachelor’s degree while 5 had a graduate diploma in social work. They had a mean of 7 years’ experience of working with psychotic patients (range, 2-12 years). By the end of the training period, only 10 participants remained. Two participants dropped out midway during the group supervision phase, one for health reasons and the other resigned from the post. The 2 participants who dropped out were the enrolled nurse and a social worker. Of the 10 trainees, 3 have attended training workshops on the clinical theories supporting cognitive therapy organised by the Beck Institute in Hong Kong. All trainees reported previous experience of providing counselling sessions to their psychotic clients residing in halfway houses using client-centred principles. None had received any formal training in client-centred therapy. None had previous experience of conducting cognitive therapy for patients with psychiatric problems prior to this study.

Patients’ Baseline Clinical Data

Ten trainees treated 10 patients through to completion of the group supervision phase (both dropouts had not recruited patients for CBT before dropping out). The patients received a mean of 12.4 cognitive therapy sessions (range, 6-19 sessions) and a mean of 26.4 weeks (range, 13-28 weeks) of treatment. They had a mean age of 47.6 (range, 29-58) years. All patients were diagnosed as suffering from an International Classification of Diseases (10th Edition) diagnosis of schizophrenia and were receiving high dose of chlorpromazine equivalents (mean dose, 653.2 mg). They were actively psychotic on recruitment with a mean score of 23.4 on the positive syndrome scale of the Positive and Negative Syndrome Scale.²⁵

Quality of Case Formulation

A total of 44 case formulations were rated for case quality (there were 24 case formulations submitted upon completion of the first phase, then 20 more case formulations upon completion of the second phase of training). The ratings made by the second author were used for comparison, as the second author was blind to the identity, the timing, and the order of the case formulations. The ratings by the first author were used for calculation of kappa agreement.

Table 1 shows the case formulation quality measures and how they changed after the group supervision phase. It is clear that, after systematic training during group number of acceptable formulations increased to 15. The change is statistically significant (Chi-square test, df = 3; p < 0.05).

Trainees’ Competence as Cognitive Behavioural Therapists

The trainees’ skills improved significantly after group supervision training over a period of 6 months. All the pre-training videotaped sessions (n = 12) submitted for evaluation were from the third session. The varying length of therapy provided by different trainees led to 7 trainees providing post-training videotaped sessions from the 10th session, while 2 trainees provided tapes of the 11th session, and one trainee taped the 9th session. A total of 22 videotaped sessions were subjected to CTRS rating.

Table 2 shows the CTRS before and after group supervision training. None of the trainees attained the level expected of competent therapists at the end of the 6-month group supervision training, though their mean skill levels improved to put them above the level of a non-specific counsellor.

Discussion

This was the first local study reporting on the value of systematically training and supervising social workers in the application of CBT for medication-resistant psychotic patients. After 10 weeks of weekly didactic lectures and 6 months of group supervision, the trainees demonstrated improvement in both general and specific therapy skills, as well as in the quality of their case formulation. A meta- analysis of 15 psychotherapy studies estimated that therapist effects account for 9% of variance in the outcome of these studies, which equates to an effect size of about 0.6. Milne and James⁶ found that supervision would lead to improved therapist competence. A local study reached a similar conclusion.⁷ Therefore, the issue of developing effective supervision strategies is an urgent research topic.

The better quality of case formulation after group supervision, but not after didactic training, suggests that didactic training may not adequately familiarise the trainees with the skills required to prepare a high-quality case formulation. This supports previous findings that experiential training is critical for effective training of psychotherapy skills, including cognitive therapy skills.

The results of the assessment of therapists’ competence upon training warrant a closer look. The trainees’ years of experience of counselling psychotic patients and their baseline results showed that their pre-training sessions were already up to the level of a supportive counsellor (CTRS total score of at least 22), with a relatively high baseline mean total CTRS score of 31.1. It is also worth noting that specific therapy skills were not absent before group supervision either, reflecting the value of the 10 weeks of didactic lectures by adding to their declarative knowledge. It is possible that trainees utilised the declarative knowledge learnt in didactic training sessions and transferred it into procedural skills (specifically technical and conceptual skills) through trying out those skills during their counselling sessions with other patients not involved in the study. It is also apparent that group supervision training did improve both general therapeutic and specific technical skills.

Nonetheless, it is rather disappointing that none of the trainees achieved the level of competent cognitive therapists defined by the CTRS. There are several possible reasons why this training programme failed to produce competent therapists. First, the supervision group might have been too large, which might restrain the time allocated for discussion of individual skills. A smaller group of 2 to 3 trainees may produce a different outcome and this awaits further research. Second, Bennett-Levy¹¹ has suggested that expertise will only develop through active self-reflection. Active self-reflection means general reflection on conceptual issues and reflection of self-schemas, attitudes, and personal beliefs. This training course did not specifically incorporate strategies to develop such a reflective capacity. Such training strategies might include reflective worksheets,²⁶ reflective writing on each session,²⁷ mindfulness training,²⁸ and emotion recognition training.²⁹ Incorporation of some of these strategies in training programmes might improve therapists’ skills in a more effective manner. Third, as discussed by Elkin,³⁰ CBT therapists have a great deal of new material to learn in their training, both in terms of specific techniques and, especially, in terms of the overall conceptualisation of cases. This is a new therapy for this group of therapies and is a psychological therapy demanding sophisticated skills that require much longer training and exposure to a greater number and variety of patients. The training issue is also complicated by the fact that their past experience and training in social work had largely involved the client- centred approach. This means novice CBT therapists with previous training in other theoretical backgrounds need to unlearn many non-CBT techniques as well. This hypothesis is partially supported by the mean baseline and final scores in general therapeutic skills and specific technical skills. A competent therapist needs a mean general therapeutic skill score of at least 24, and a mean specific technical skill score of at least 20. As evident in Table 2, it is clear that the trainee therapists had reached a baseline mean score of 20, and their final score, 22 was approaching a competent level. However, their baseline mean specific technical skill score was only 10, and this reached only 13 after group supervision training. That is a score still far below the required specific technical skill score of 20 needed for competence. It is important to be aware that this hypothesis is speculative. Most CTRS items load on one single factor, suggesting that the subscores and individual item scores are not differentiating particular skills, thus only the total CTRS score is a good measure of overall competency in cognitive therapy.²³ Therefore, specific CTRS technical skill sub- scores may not accurately reflect specific CBT techniques. The magnitude of change in the 2 subscales before and after training can best be viewed as a crude indicator of change in competency in the 2 skill areas (general therapeutic skills and specific technical skills).

Cognitive therapy is a time-limited, highly structured therapy requiring a tactful balance of directive guidance and Socratic dialogue. Client-centred therapy emphasises warmth, empathy, and unconditional acceptance, which purports to provide an optimal therapeutic environment for a client’s personal growth and recovery. There is little emphasis on promoting change whereas cognitive therapy emphasises various cognitive and behavioural skills that promote change. Teaching and coaching such skills may at times require a minor therapeutic relationship strain, especially when patients are reluctant to move out of their ‘comfort zones’ to try new behavioural experiments (that is, temporarily abandon old behaviours and try out newly learned behaviours). Trainees with a client-centred therapy background have found this therapeutic alliance rupture unacceptable and counter-intuitive. Yet there is literature suggesting that minor therapeutic ruptures, if skilfully repaired, will speed up therapeutic progress.²⁸

This study has a number of limitations. First, the sample size is small and restricted to the staff from one single mental health organisation so is not generalisable. Nevertheless, this is a pilot study that aims to stimulate further interest in the study of the supervision process and outcome and is by no means conclusive. Second, as there is no control group without supervision training, the improvement in declarative and procedural therapist skills may have been due to other factors like an increase in work experience over time. However allocation of a group of patients under the care of novice therapists without proper supervision may be considered ethically unacceptable. Third, the assessments of therapist competence and quality of case formulation were conducted by the trainer himself and were not blind. The rater’s bias due to the expectancy effect could not be ruled out. This weakness was due to the lack of experienced supervisors familiar with the use of the CTRS and rating of quality of case formulation. Finally, the measurement of therapist competence with the CTRS and quality of case formulation was relatively restricted and failed to measure other important therapist skills like self-reflection skills or interpersonal perceptual skills. Satisfaction with the training content and style was also not measured. In order to address this limitation, a qualitative study is now under way to understand the trainees’ attitudes towards the supervision experience.

Acknowledgements

The first author would like to thank all the trainee therapists from the Richmond Fellowship of Hong Kong for participating in the training and supervision. The authors would also like to thank all the patients involved in this study. Finally the first author would also like to thank Dr Timothy Chan, Director of the Richmond Fellowship of Hong Kong, for his support of this study.

References

1. National Institute for Clinical Excellence. Core interventions in the management of schizophrenia in secondary and primary care. London: NICE; 2003.
2. Tarrier N, Wykes T. Is there evidence that cognitive behaviour therapy is an effective treatment for schizophrenia? A cautious or cautionary tale? Behav Res Ther 2004;42:1377-401.
3. Ng RM, Cheung M, Suen L. Cognitive behavioural therapy for psychosis: an overview and its application in Hong Kong. Hong Kong J Psychiatry 2003;13:26-33.
4. Tarrier N. Cognitive behaviour therapy for schizophrenia — a review of development, evidence and implementation. Psychother Psychosom 2005;74:136-44.
5. Liese BS, Beck J. Cognitive therapy supervision. In: Watkins EC, editor. Handbook of psychotherapy supervision. New York: Wiley; 1997:114-33.
6. Milne D, James I. A systematic review of effective cognitive- behavioural supervision. Br J Clin Psychol 2000;39:111-27.
7. Ng RM. Cognitive therapy supervision: a local experience and pilot study. Hong Kong J Psychiatry 2005;15:122-6.
8. Binder JL. Issues in teaching and learning time-limited psychodynamic psychotherapy. Clin Psychol Rev 1999;19:705-19.
9. Lambert MJ, Ogles BM. The effectiveness of psychotherapy supervision. In: Watkins EC, editor. Handbook of psychotherapy supervision. New York: Wiley; 1997:421-46.
10. Kingdon D, Turkington D. Cognitive therapy for schizophrenia. New York: Guildford; 2005.
11. 1 Bennett-Levy J. Therapist skills: a cognitive model of their acquisition and refinement. Behav Cognitive Psychother 2006;34:57-78.
12. Kingdon D, Turkington D. Cognitive behaviour therapy for schizophrenia. Hillsdale, NJ: Lawrence Erlbaum Associates; 1994.
13. Tarrier N, Calam R. New development in cognitive-behavioural case formulation: epidemiological, systemic and social context: an integrative approach. Behav Cognitive Psychother 2002;30:311-28.
14. Wampold BE, Mondin GW, Moody M, Stich F, Benson K, Ahn HN. A meta-analysis of outcome studies comparing bona fide psychotherapies: empirically, “All must have prizes”. Psychol Bull 1997;122:203-15.
15. Eells TD, Lombart KG, Kendjelic EM, Turner LC, Lucas CP. The quality of psychotherapy case formulations: a comparison of expert, experienced, and novice cognitive-behavioral and psychodynamic therapists. J Consult Clin Psychol 2005;73:579-89.
16. Eells TD. Psychotherapy case formulation: history and current status. In: Eells TD, editor. Handbook of psychotherapy case formulation. New York: Guildford; 1997:1-25.
17. Bieling PJ, Kuyken W. Is cognitive case formulation science or science fiction? Clin Psychol Sci Prac 2003;10:52-69.
18. Kuyken W, Fothergill CD, Muse M, Chadwick P. The reliability and quality of case formulation. Behav Res Ther 2005;43:1187-201.
19. Morrison AP, editor. A casebook of cognitive therapy for psychosis. East Sussex: Brunner-Routledge; 2002.
20. Freeman D, Garety P. Cognitive therapy for an individual with long- standing persecutory delusion: incorporating emotional processes into a multi-factorial perspective on delusional beliefs. In: Morrison AP, editor. A casebook of cognitive therapy for psychosis. East Sussex: Brunner-Routledge; 2002:173-96.
21. Morrison AP. Cognitive therapy for drug-resistant auditory hallucinations: a case example. In: Morrison AP, editor. A casebook of cognitive therapy for psychosis. East Sussex: Brunner-Routledge; 2002:132-47.
22. Dobson KS, Shaw BF, Vallis TM. Reliability of a measure of the quality of cognitive therapy. Br J Clin Psychol 1985;24:295-300.
23. Vallis TM, Shaw BF, Dobson KS. The Cognitive Therapy Scale: psychometric properties. J Consult Clin Psychol 1986;54:381-5.
24. Hollon SD, DeRubeis RJ, Evans MD, Weimer MJ, Garvey MJ, Grove WM, et al. Cognitive therapy and pharmacotherapy for depression. Singly and in combination. Arch Gen Psychiatry 1992;49:774-81.
25. Kay SR, Fiszbein A, Opler LA. The positive and negative syndrome scale (PANSS) for schizophrenia. Schizophr Bull 1987;13:261-76.
26. Bennett-Levy J, Thwaites R. Self and self-reflection in the therapeutic relationship: a conceptual map and practical strategies for the training, supervision and self-supervision of interpersonal skills. In: Gilbert P, Leahy RL, editors. The therapeutic relationship in the cognitive behavioral psychotherapies. London: Routledge; 2007:255-82.
27. Bolton G. Reflective practice. 2nd ed. London: Sage; 2002.
28. Safran JD, Muran JC. Negotiating the therapeutic alliance: a relational treatment guide. New York: Guildford; 2000.
29. Machado PP, Beutler LE, Greenberg LS. Emotion recognition in psychotherapy: impact of therapist level of experience and emotional awareness. J Clin Psychol 1999;55:39-57.
30. Elkin I. A major dilemma in psychotherapy outcome research: disentangling therapists from therapies. Clin Psychol Sci Prac 1999;6:10-32.