



































Interpreter-Mediated Therapy for Refugees:


Graduate Student Journal of Psychology          Copyright 2011 by the Department of Counseling and Clinical Psychology 

2011, Vol. 13               Teachers College, Columbia University 

17 

 

 

Naturalistic Examination of a Training Clinic: Is There a Relationship Between 

Therapist Perception and Client Self-Report of Treatment Outcomes? 
 

Scott H. Waltman, Katherine H. Rex, and Alyson Williams 
Pacific University 

Numerous psychometrically sound measures of psychotherapy outcome have been developed; 

however, few clinicians administer such tools, and when they do so, they rarely utilize these results 

to inform treatment.   Moreover, studies have reported a discrepancy between therapists’ judgment 

and clients’ self-report of treatment outcome.  The current study aimed to further investigate the 

relationship between therapists’ judgment and clients’ self-report of treatment outcome in a 

psychological training clinic, using the criteria of reliable change (RC) and clinically significant 

change (CSC).  One hundred and forty-four clients receiving therapy at a psychological training 

clinic in the Pacific Northwest were administered the Outcome Questionnaire 45.2 (OQ) at the 

beginning and end of treatment.  Study participants were primarily Caucasian (88%) and female 

(56%), with a mean age of 32 years.  Changes in outcome scores were compared to therapist 

judgments regarding treatment effectiveness.  Results indicated a significant relationship between 

therapists’ judgment of treatment outcome and whether clients achieved RC or CSC.  A large effect 

size (Φ = .56) was found between CSC and therapist judgment whereas a medium effect size (Φ = 

.32) was found between RC and therapist judgment.  These results suggest that there is lack of 

agreement between the RC and CSC criteria, and that outcome measure alone cannot account for 

therapist judgment about treatment outcomes.  Suggestions are made for maximizing the accuracy 

and practicality of client outcome assessment. 

 

 

A number of psychometrically sound instruments 

assessing psychotherapy outcome exist; however, the 

majority of clinicians do not use such tools (Lambert & 

Hawkins, 2004).  Moreover, even when clinicians do 

administer measures that track therapeutic outcomes, many 

do not utilize the results to inform treatment (Garland, Kruse, 

& Aarons, 2003).

 Research findings show discrepancies 

between therapists’ judgment and clients’ self-report of 

treatment outcomes (Swift, Callahan, & Levine, 2009); it 

appears that many clinicians evaluate treatment outcomes 

based on their own perceptions, rather than based on 

standardized measures intended to accurately assess clients’ 

clinical symptoms and functioning. This raises important 

clinical questions, given that the accurate evaluation of 

treatment outcomes informs whether treatment was effective 

and is a critical component of improving treatment (American 

Psychological Association [APA], 2006).  

In a recent study of a psychological training clinic, Swift 

and colleagues (2009) compared two methods of evaluating 

treatment outcome: therapists’ judgment of treatment 

outcome and clients’ scores on standardized outcome tracking 

measures.  The authors reported a low level of agreement 

                                                 
Scott H. Waltman, M.S., Katherine H. Rex, M.S., and Alyson 

Williams, Ph.D., School of Professional Psychology, Pacific 

University.  

Scott Waltman would like to thank Dr. Alyson Williams for her 

patience, guidance, and assistance in that endeavor.  

Correspondence concerning this article should be addressed to 

Scott Waltman, School of Professional Psychology, Pacific 

University, 190 SE 8th Ave. Hillsboro, OR  97123. Email: 

walt2155@pacificu.edu. 

between these two methods, suggesting that therapists who 

rely on their judgment alone are more likely to describe 

treatment as being successful compared to those using 

collateral information from objective outcome measures.  

Drawing on these research findings by Swift and colleagues 

(2009), the current study aims to investigate whether there is 

a lack of agreement between therapists’ judgment of 

treatment outcomes and clients’ self-reported changes in other 

psychological training clinics. The present study uses a 

naturalistic design (i.e., absence of experimental 

manipulations) that provides a realistic evaluation of how a 

training clinic functions. 

 

Clinicians’ Use of Outcome Measurements 

 

Given that therapists are subject to the same judgment 

errors as all other humans (Ruscio, 2007), numerous 

treatment outcome measures have been developed to help 

clinicians assess the effectiveness and quality of psychosocial 

interventions (Garland et al., 2003; Hatfield & Ogles, 2004; 

Mours, Campbell, Gathercoal, & Peterson, 2009).  Usually, 

these are client self-report measures that assess on a weekly 

timeframe the impact of presenting problems on functioning.  

However, it is estimated that less than one-third of all 

clinicians use outcome tracking measures in their clinical 

practice (Lambert & Hawkins, 2004). Underuse of outcome 

measures was reported by Garland and colleagues (2003) in a 

survey of clinicians who participated in a state program that 

mandated the use of outcome measures.  The authors found 

that as many as 92% of the clinicians had never referenced 

standardized outcome measures in their practice (Garland et 



WALTMAN, REX, & WILLIAMS 

 

18 

 

al., 2003).  In a different survey of a national sample of 

licensed psychologists, only 37% of respondents reported 

using some sort of standardized assessment in their practice 

(Hatfield & Ogles, 2004).  

It is necessary to examine therapist attitudes about 

outcome measures, as this may illuminate why these 

measures are so infrequently utilized.  In the study by Garland 

and colleagues (2003), clinicians reported that outcome 

measures are cumbersome or intrusive, can be expensive to 

practitioners, are burdensome to clients, and do not provide 

new information (Garland et al., 2003).  Rather than using 

outcome measures that have been shown to lead to 

improvements in treatment outcomes (Reese, Norsworthy, & 

Rowlands, 2009) and increased cost-effectiveness (Slade et 

al., 2006), many of these clinicians reported using “real world 

functional indicators” (e.g., school grades), clinical intuition, 

and client-reported satisfaction in order to evaluate treatment 

effectiveness (Garland et al., 2003).  It is clear that some 

therapists may view their clinical judgment as being more 

accurate or useful than the information obtained from 

outcome measures.  It is unclear, however, whether this trend 

also exists in a training clinic setting, as there is limited 

research on the relationship between student clinician and 

licensed professional judgment. Understanding the 

relationship between therapist judgment about treatment 

outcomes and client changes in scores on outcome measures 

will be informative as to whether some student therapists 

view their developing clinical judgment as being more 

accurate and useful than the information generated from 

standardized outcome measures. 

 

Measurement of Reliable Change and Clinically 

Significant Change 

The regular use of standardized outcome measures is a 

recommended routine practice for tracking client progress 

(Hatfield & Ogles, 2004).  Advantages of using these 

measures include their established reliability and validity and 

their sensitivity in determining clinically meaningful changes 

in client functioning.  The concepts of reliable change (RC) 

and clinically significant change (CSC) were introduced as a 

statistical approach of determining when meaningful change 

has occurred (Jacobson, Follette, & Revenstorf, 1984).  RC 

indicates that a change in scores on a measure is sufficiently 

larger than the standard error of the difference between the 

two scores, and thus is not due to chance (Jacobson & Truax, 

1991).  CSC is more a more stringent criterion for 

determining treatment outcome than RC, as CSC requires that 

RC has occurred and that a client’s score on an outcome 

measure has moved from the dysfunctional to the functional 

range (Jacobson & Truax, 1991).  In order for clients to 

obtain CSC, they must begin treatment in the clinical range of 

functioning and move into the nonclinical range of 

functioning (Wise, 2004).  The use of the criteria of RC and 

CSC has advantages over other statistical methods.  A 

traditional statistical method of comparing different groups is 

comparing group means (e.g., analysis of variance; ANOVA), 

which has two major limitations (Jacobson, Roberts, Berns, & 

McGlinchey, 1999).  First, comparing group means provides 

little information about the variability within those groups 

(i.e., the proportion of group members who improved or 

recovered).  Second, a large effect size does not mean that the 

effect was clinically meaningful (Jacobson et al., 1999).  

The criteria of RC and CSC have been used to evaluate 

the effectiveness of treatment in psychological training 

clinics.  Callahan and Hynan (2005), for example, examined 

the treatment outcomes of 61 clients who were treated in a 

psychological training clinic by comparing clients’ changes in 

scores on a routinely administered measure, the Outcome 

Questionnaire 45.2 (OQ; Lambert et al., 2004), from the 

beginning to the end of treatment.  They determined that over 

the course of treatment, 18% of participants achieved CSC, 

33% demonstrated RC, and 67% showed no RC (Callahan & 

Hynan, 2005).  This study illustrated that fewer clients 

achieve CSC than RC, which is to be expected as CSC is a 

more stringent criterion to meet. In another study of a training 

clinic, Swift and colleagues (2009) found a CSC rate of 23% 

and a RC rate of 37%.  The authors concluded that therapists 

often labeled treatment as successful when CSC had not 

occurred, and sometimes therapists labeled treatment as 

unsuccessful when CSC had happened (Swift et al., 2009). 

 

Therapist Judgment of Treatment Outcome 

 

Borrowing from the premature termination literature, it 

has been suggested that therapist judgment may be the best 

method of determining when a client has prematurely 

terminated (Wierzbicki & Pekarik, 1993); however, a 

problem with using therapist judgment to determine outcome 

is the issue of reliability (Wierzbicki & Pekarik, 1993).  Not 

all therapists may decide a client has prematurely terminated 

from therapy based on the same criteria (Hatchett & Park, 

2003; Swift et al., 2009), and research has found that 

therapists are likely to use self-serving attributions when 

explaining why a client prematurely terminated from 

treatment (Murdock, Edwards, & Murdock, 2010).  

Additionally, therapists’ and client’s descriptions of treatment 

termination may differ (Hunsley, Aubry, Verstervelt, & Vito 

1999).  In fact, it has been proposed that therapist perception 

of progress and client self-report of satisfaction have a low 

level of agreement (Mours et al., 2009).  Several studies have 

failed to find a significant relationship between client 

satisfaction and symptomatic improvement (Lambert, Okiishi, 

Finch, & Johnson, 1998; Lunnen & Olges, 1998; Lunnen, 

Ogle, & Pappas, 2008; Pekarik & Wolff, 1996), and yet, 

multiple surveys of practicing clinicians have found that 

many clinicians are likely to rely on their own intuition and 

the client’s self-report to determine if treatment has been 

successful (Garland et al., 2003; Mours et al., 2009). 

Given potential discrepancies between therapists’ and 

clients’ reports, treatment evaluation based on therapists’ 

judgment alone may be insufficient.  Research has 

demonstrated that therapists’ judgments about treatment 

outcome are more likely to be accurate only when it is 

positive (Hunsley et al., 1999).  This may be due to biases in 



TRAINING CLINIC EXAMINATION 

 

19 

 

the judgment of the clinicians.  Specifically, therapists may be 

more likely to attend to positive information about treatment 

outcomes than to negative information (Ruscio, 2007).  These 

biases in therapist judgment are problematic, especially since 

it is likely that they decrease therapists’ ability to recognize 

the occurrence of a negative treatment outcome. 

The current study is an investigation of the relationship 

between clients’ changes in Outcome Questionnaire-45.2 

scores (OQ; Lambert et al., 2004) and student therapist 

judgment about treatment success. This study is particularly 

relevant because the student clinicians had access to the OQ 

data as they rated the successfulness of treatment. Thus, the 

level of agreement between therapist judgment and outcome 

scores may be interpreted as an indicator of how influential 

outcome measurement is to student therapist judgment. The 

following research questions guided the present examination 

(a) how often do clients achieve RC and CSC?; and (b) what 

is the relationship between therapist judgment of treatment 

outcome and client changes in OQ scores? It was 

hypothesized that therapist judgment about treatment 

outcomes would be related to whether RC and CSC occur on 

client OQ scores. 

 

Method 

 

Participants 

Clients.  Archival data were used from 144 clients 

receiving therapy during the 2007-2008 academic year at a 

university-based psychological training clinic located on a 

small campus in a downtown urban area of the Pacific 

Northwest.  This training clinic provides outpatient services 

to the general public.  The clinic operates on a sliding fee 

scale, with session fees ranging from $20 to $85 depending 

on client income level.  The clinic additionally offers a 

discounted rate of $20 per session for college students.  

Although data concerning socioeconomic status were not 

collected as a part of this study, the clients receiving therapy 

from this clinic are often unemployed or working poor (i.e., 

underemployed) (Brooks, 2007).  Clients may receive 

individual or couples therapy, and may present with a variety 

of presenting concerns.  Typical presenting problems include 

depressive symptoms, anxiety symptoms, and relationship 

concerns.  Individuals interested in treatment participate in a 

standardized screening procedure.  Those who endorse 

particular risk factors, such as active suicidal ideation or 

excessive substance use concerns, are referred out to 

treatment settings that may provide higher levels of care.  

Of the 144 clients, 31% (n = 45) were excluded from 

analysis due to either missing OQ test score data or the client 

having only attended an intake session.  Clients who were 

identified as only attending an intake session were not 

included in this evaluation, as it has been suggested that 

clients who did not return for therapy after the intake did not 

start treatment (Hatchett & Park, 2003).  Furthermore, at least 

two administrations of an outcome measure are required to 

calculate RC and CSC.  Of the remaining clients, 

demographic data was available for 67% (n = 66).  This 

sample of 66 clients had more women (56.1%) than men 

(43.9%), had an average age of 32 years (SD = 8.87), with a 

range of 17 to 57 years.  In regards to ethnicity, 87.9% of 

clients were Caucasian, 4.5% were of Asian descent, 1.5% 

were African-American, 4.5% were multi-ethnic, and 1.5% 

were unknown.  Inclusion in the current study depended upon 

the presence of both a therapist rating of the successfulness of 

treatment and test scores from the first and last 

administrations of the OQ. 

A Pearson’s Chi-Square (χ
2
) was utilized to determine if 

there was a relationship between therapist description of 

treatment outcome and whether or not demographic data were 

available.  Additionally, an ANOVA was used to evaluate 

whether group differences existed in the average amount of 

change in OQ scores depending on the availability of 

demographic data.  No significant differences were found in 

these analyses, which suggest that it was appropriate to 

include clients with missing demographic information in 

further analyses.  

Therapists.  Clients received therapy from doctoral 

students who were enrolled in a clinical psychology program 

and supervised by licensed clinical or counseling 

psychologists.  These clinicians were trained in treatment for 

presenting problems ranging from anxiety and depression to 

relationship problems.  Supervisors were licensed 

psychologists who were either faculty members or 

psychologists from the community with private practices.  

This training program follows a practitioner-scholar model, 

and student therapists were either in their second, third, or 

fourth year in the program.  Three of these therapists were 

pre-doctoral interns who were either in their fifth or sixth year 

of training.  In total, approximately 45 therapists recorded 

data for this study.  At the time this data were collected, 

information about individual therapists was not collected due 

to concerns about student privacy and confidentiality.  This 

will be discussed later as a limitation of the study.  

 

Measures 

Outcome Questionnaire.  The OQ is a 45-item self-

report measure that assesses general psychological distress 

(Lambert et al., 2004).  An overall score may be calculated 

after administration, along with subscale scores related to 

symptom distress, interpersonal functioning, and social role.  

The OQ is typically administered to every adult client at 

every session in this training clinic.  For this study, only the 

first and last administrations of the OQ were analyzed.  The 

average number of sessions completed by clients was 13. 

Other studies have used the OQ to measure RC and CSC, and 

to identify when individual treatment has been successful 

(Callahan & Hynan, 2005; Callahan, Swift, & Hynan, 2006).  

On the OQ, an overall score above 63 is indicative of 

symptoms in the clinical range (Anderson & Lambert, 2001).  

The OQ manual states that no gender differences exist 

between male and female scores.  The manual also specifies 

that no significant differences have been found between the 

total scores of Caucasians, Hispanics, and African-

Americans.  The OQ has adequate psychometric properties, 



WALTMAN, REX, & WILLIAMS 

 

20 

 

with a test-rest reliability of .87 and internal consistency of 

.93 (Lambert et al., 2004).  The internal consistency for this 

sample could not be calculated, as only the total OQ scores 

from the first and last administration were analyzed in this 

study. 

Closing File Outcome Tracking Form.  The Closing 

File Outcome Tracking Form (interested readers may contact 

the corresponding author for a copy; Brown, Williams, 

Waltman, & Sutton, 2010) was created for use in this 

particular training clinic.  It is a brief form that is completed 

by the client’s assigned therapist as a part of the regular file 

closing procedure.  This form contains information about 

diagnosis, client scores on the OQ, therapist description of 

therapy outcome, therapy duration, and demographic 

information.  For this study, only a subset of information 

from the form was analyzed: data related to OQ scores from 

the first and last administration of the measure, therapist 

description of the outcome of therapy, and demographic 

information. 

 

Procedure  
Clients were informed that their demographic and 

treatment related information may be used for research 

purposes during the informed consent process at the 

beginning of therapy. Approval for the study was obtained 

from Pacific University’s Institutional Review Board (IRB).  

Data obtained from the Closing File Outcome Tracking Form 

were de-identified and coded into a database.  Data were 

coded into the following three variables: therapist judgment, 

reliable change, and clinically significant change.  Pearson’s 

Chi-Square (χ
2
) was utilized to determine if there was a 

relationship between therapist description of treatment 

outcome and client changes in OQ scores.  A phi correlation 

(Φ) was conducted to measure the relationship between 

changes in OQ scores and therapist rating of treatment 

successes.  

Therapist judgment.  The information about therapist 

judgment of the treatment outcome was recorded on the 

Closing File Outcome Tracking Form by each student 

therapist.  Therapist judgment was coded as it was recorded 

on the form: “successful completion of treatment,” 

“substantial progress without successful completion of 

treatment,” “incomplete or moderate progress,” or “no 

progress.” 

Reliable change and clinically significant change.  

Based on the recommendations of the test publisher (Lambert 

et al., 2004), a decrease in total OQ scores of 14 or more was 

coded as “reliable change.”  A change in total OQ scores of 

13 or less was coded as “no reliable change.”  Clinically 

significant change requires that RC has occurred and that a 

predetermined cut-off point has been crossed.  A drop in total 

OQ score of 14 or more, and a decrease from a total score of 

above 64, to a total score below 63, was coded as “clinically 

significant change.”  A failure to attain a drop in OQ scores of 

at least 14 points or a failure to cross the cutoff point of 63 

was coded as “no clinically significant change.”  The 47 

clients who began treatment with OQ scores below 64 were 

unable to achieve CSC and were thus excluded from the 

examination of the relationship between therapist judgment 

and CSC. 

 

Results 

 

Descriptive Statistics 

Descriptive statistics were used to compute the frequency 

of RC, CSC, and therapist description of treatment outcomes.  

Overall, from the beginning to the end of treatment, the mean 

change in total OQ score for the entire sample was a 9.67 

point decrease.  Only 35.4% of clients achieved RC.  In 

determining CSC, it was found that only 18.2% of the clients 

had achieved CSC.  It is worth noting that 47.5% of the 

clients began treatment with an OQ score in the nonclinical 

range, so they could not achieve CSC; of the clients who 

began treatment in the clinical range of functioning, 34.6% 

achieved CSC.  It was found that therapists described 10.1% 

of treatment outcomes as “no progress,” 33.3% of treatment 

outcomes as “incomplete or moderate progress,” 21.2% of 

treatment outcomes as “substantial progress without 

successful completion of treatment,” and 35.4% of treatment 

outcomes as “successful completion of treatment.”  Table 1 

provides the percentages of treatment outcomes that achieved 

RC or CSC and the percentage of treatment outcomes 

according to therapist description. 

 

 

Table 1 

 

Comparison of Methods for Determining Client Treatment Outcome 

Therapist Judgment of Treatment 

Outcome 

Total
a 

(%)
 

  

Reliable Change  Clinically Significant Change
b 

Yes (%) No (%) Yes (%) No (%) 

Successful Completion  35.4      18.2 17.2  8.1 27.3 

Substantial Progress  21.2 9.1 12.1  6.1 15.2 

Incomplete or Moderate Progress 33.3 7.1 26.3  4.0 29.3 

No Progress 10.1 1.0 9.1  0 10.1 

Total       35.4 64.6  18.2 81.8 
an = 99. b47.5% of clients began treatment with OQ scores in the nonclinical range, and subsequently could not achieve clinically significant 

change. 



TRAINING CLINIC EXAMINATION 

 

21 

 

Therapist Judgment, Reliable Change, and Clinically 

Significant Change 

 

 

 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 
 

 

In order to evaluate if therapist judgment was related to 

RC, a two-way contingency table analysis was conducted.  

The two variables were therapist judgment with four levels 

(i.e., “successful completion of treatment,” “substantial 

progress without successful completion of treatment, 

“incomplete or moderate progress,” or “no progress”) and RC 

with two levels (i.e., RC and no RC). The two variables, 

therapist judgment and reliable change were found to be 

significantly related, χ
2
(3, N = 99) = 10.18, p =.017, Φ = .321, 

consistent with a medium effect size (Cohen, 1988).  These 

results support the conclusion that therapist judgment of 

treatment outcome is related to whether RC occurs. 

In order to determine if therapist judgment was related to 

whether CSC occurred, a second two-way contingency table 

analysis was completed.  The two variables were therapist 

judgment with four levels (i.e., “successful completion of 

treatment,” “substantial progress without successful 

completion of treatment,” “incomplete or moderate progress,” 

or “no progress”) and CSC with two levels (i.e., CSC and no 

CSC).  The two variables of therapist judgment and CSC 

were found to be significantly related, Pearson χ
2
(3, N = 52) = 

16.20, p =.001, Φ = .558, consistent with a large effect size 

(Cohen, 1988).  These results support the conclusion that a 

relationship exists between therapist judgment of treatment 

outcome and whether CSC occurs. 

 

Post-Hoc Analysis 

 After it was found that a sizeable portion of the sample 

was not capable of achieving CSC, the researchers decided to 

conduct an additional post-hoc analysis, a one-way analysis 

of variance (ANOVA), to evaluate the relationship between 

therapist judgment of treatment outcomes and the change in 

OQ scores from the first to last administration of the measure.  

It was hypothesized that significant differences would be 

found between the average amount of change in total OQ 

scores and therapists’ judgment of treatment outcomes.  The 

independent variable was therapist judgment and the 

dependent variable was the change in OQ scores from first to 

last administration of the measure.  The results of the 

ANOVA were significant, F(3, 95) = 4.41, p = .001.  The 

strength of the relationship between therapist judgment and 

the change in OQ scores, assessed by η
2
, was moderate, with 

the therapist judgment accounting for 12% of the variance in 

the dependent variable.  

Follow up tests were conducted to evaluate pairwise 

differences among the means.  Based on Levene’s test of the 

homogeneity of variance, it can be assumed that the variances 

were homogenous among the three groups.  Post hoc 

comparisons were therefore conducted using Tukey’s HSD  

There was a significant difference in the means between 

the group that was judged to have successfully completed 

treatment (M = -18.80, SD = 23.13) and the group that was 

judged to have made no progress (M = 1.50, SD = 10.64), as 

well as between the group that was judged to have 

 

successfully completed treatment and the group that was 

judged to have incomplete or moderate progress (M = -2.55, 

SD = 21.12).  No significant difference was found between 

the substantial progress without completion of treatment 

group (M = -10.95, SD = 20.87), and the no progress group.  

There was not a significant difference found between the 

other pairwise comparisons.  

 

Discussion 

 

The current study investigated the relationship between 

therapists’ judgment and clients’ self-report of treatment 

outcome; the latter was operationalized as changes in 

outcome scores using both RC and CSC criteria.  The study is 

of particular relevance to those interested in understanding 

how the use of outcome measures affects therapist judgment, 

as the therapists had access to the OQ data when they were 

rating the overall successfulness of treatment.   

The results of this study demonstrate that a relationship 

exists between therapists’ judgment of treatment outcome and 

whether a client achieves RC or CSC on an outcome measure.  

Additionally, a large effect size was found between CSC and 

therapist judgment, indicating a strong relationship between 

these two variables.  These findings suggest that training 

therapists often labeled treatment as successful when CSC did 

occur, which is commensurate with findings from a prior 

study by Swift and colleagues (2009).  Although these results 

denote that therapist judgment and RC are related, the results 

also suggest that there is a low level of agreement between 

these two methods of determining treatment outcomes as 17% 

of therapists labeled treatment as successful when RC had not 

occurred, and 8% labeled treatment as unsuccessful when RC 

had occurred.  Further research is needed to explore why this 

low level of agreement exists.  It is possible that this low level 

of agreement is due to clinicians trusting their own judgment 

more than the outcome tracking measure. 

The post hoc analysis revealed that changes in OQ scores 

significantly differed between the different client groups 

depending on whether the therapist judged treatment to be 

Table 2  

 

Mean Change in Total OQ score by Therapist Judgment of 

Treatment Outcome
 

Therapist 

Judgment  

Total 

Score 
M (SD) 95% CI

a
 

Successful 

Completion 
35 -18.80 (23.13) -25.87, -11.73 

Substantial 

Progress 
21 -10.95 (20.87) 

-20.08, -1.82 

 

Incomplete or 

Moderate Progress 
33 -2.55 (21.12) 

-9.83, 4.73 

 

No Progress 10 1.50 (10.64) -11.7, 14.73 

Total 99 -9.67 (22.15)   
aCI = confidence interval. 

 



WALTMAN, REX, & WILLIAMS 

 

22 

 

successful or incomplete (See Table 2).  There were no 

significant differences between the successful completion and 

substantial progress groups, suggesting that the distinction 

between these different groups may not be meaningful.  

Within the subset of clients who were judged to have 

successfully completed treatment, the mean change in OQ 

score was a decrease of 18.8 points.  This supports the 

conclusion that a client who has been judged to have 

successfully completed treatment will, on average, have had a 

decrease in OQ score by 18.8 points, which is a larger score 

decrease than the one necessary for obtaining RC.  However, 

this analysis does not provide information about the level of 

agreement between therapist judgment and RC as it applies to 

individual cases. 

The rates of RC and CSC found in this study seem to be 

commensurate with those reported in other psychological 

training clinics. In this study, it was found that 35.4% of 

clients achieved RC.  Prior reported rates of RC in other 

training clinics include 37% (Swift et al., 2009) and 33% 

(Callahan & Hynan, 2005).  In this study, it was established 

that 18.2% of clients achieved CSC; it is worth noting, 

however, that 47.5% of the clients included in this study 

could not achieve CSC because they started treatment with a 

score in the nonclinical range.  Prior reported rates of CSC in 

other training clinics include 23% (Swift et al., 2009) and 

18% (Callahan & Hynan, 2005).  As such, the rates of RC and 

CSC found in this study appear consistent with the rates 

reported by other training clinics. 

 

Rates of Reliable and Clinically Significant Change 

It has been suggested that therapist perception of 

treatment progress and client self-report of treatment 

satisfaction are not accurate indicators of treatment 

effectiveness (Mours et al., 2009).  By contrast, this study 

showed that therapist perception of treatment outcome was 

significantly related to the criteria of RC and CSC on a 

standardized measure.  In this study, therapists judged that 

56.6% of clients achieved successful completion of treatment 

or made substantial progress, whereas 35.4% of clients were 

found to have met the criteria for RC.  When comparing 

treatment success rates, there was a discrepancy of over 20%, 

which was potentially due to errors in therapists’ perception 

(Ruscio, 2007).  

It is also possible that in some cases, treatment was 

successful despite the client not achieving RC on the total OQ 

score.  For this particular subgroup, the mean change in total 

OQ scores was a decrease by 9.67 points.  This decrease in 

overall score could indicate RC on one of the subscales of the 

OQ, but this could not be further investigated since subscale 

data were not collected for the database.  The OQ has three 

subscales: Symptom Distress, Interpersonal Relations, and 

Social Role.  On these subscales, a respective decrease in 

score of 10, 8, and 7 points is considered RC (Lambert et al., 

2004).  It is possible that if subscale data were collected, then 

individuals who did not demonstrate RC on the OQ may 

demonstrate RC on one of these subscales.  The therapists 

who rated the successfulness of treatment were cognizant of 

this subscale data, and that knowledge may, in part, account 

for the disparity between therapist judgment and RC rates.  

As previously stated, OQ scores were the only client outcome 

data collected in this study.  It is therefore possible that if 

these clients were administered a standardized measure 

specific to their presenting difficulties, then perhaps RC may 

have been found. In many instances in this clinic, additional 

standardized measures were utilized.  For example, clients 

being treated for an anxiety disorder are often administered an 

inventory specific to anxiety, such as the Beck Anxiety 

Inventory (BAI; Beck & Steer, 1990).  If a client were 

repeatedly administered an additional measure, then scores 

from that measure may have influenced the therapists’ 

description of the outcome of therapy; however, data from 

any additional measures were not collected as a part of this 

investigation.  The knowledge gained from additional 

assessments may account for some of the disparity in 

treatment success rates.  

 

Implications for the Relationship between Therapist 

Judgment and Outcome Measures 

The study’s findings of the low level of agreement 

between changes in scores on the OQ and therapist judgment 

have important implications for clinical training and practice. 

Student therapists should be trained to recognize that their 

perception of client progress may not accurately reflect the 

client’s self report.  Psychologists in training may, in some 

cases, be selectively attending to that information which 

confirms their beliefs that the client is making progress, 

thereby ignoring any conflicting information (Ruscio, 2007).  

For example, if a therapist feels that a client is making 

progress, then he or she may focus on how the client 

consistently attends sessions and ignore the client’s self-

report of increasing symptomatology on an outcome measure.  

Clinicians would benefit from seeking information that is 

both confirmatory and disconfirmatory to their perceptions 

(Ridley, 2005).  

Student clinicians’ supervisors may play a role in helping 

students learn how to obtain such information and to navigate 

the process of developing a comprehensive view of clients’ 

progress in treatment.  The use of standardized outcome 

measures can and should be a preferred method to evaluate 

the accuracy of a clinician’s intuition, and to inform practice 

by tracking client progress.  Psychologists in training are 

forming the habits which will guide their future practice, and 

it is imperative that they form habits consistent with best and 

ethical practice guidelines (APA, 2006).  Such habits would 

include regularly using outcome tracking measures to track 

client progress, evaluating if changes in course of treatment 

are necessary, and determining if therapy was successful.  

This study utilized a measure of general distress, but other 

outcome tracking measures which quantify symptom distress, 

life satisfaction, and personality functioning are also valuable 

sources of information.  In addition to seeking guidance from 

supervisors, student clinicians may benefit from participation 

in trainings designed to introduce the concept of using 



TRAINING CLINIC EXAMINATION 

 

23 

 

multiple sources of information to evaluate treatment 

progress. 

 

Limitations 

The present study used archival data, which therefore 

limited the type of data available for analysis.  The 

information about therapist judgment of outcome was 

categorical, limiting the types of analyses which could be 

conducted.  Also, this study utilized nonparametric statistics, 

and it can therefore be difficult to estimate how the findings 

of this study relate to the general population.  This study took 

place in the Pacific Northwest with a sample of clients that 

may not represent the cultural or ethnic diversity of other 

regions.  Of the 99 clients whose information was included 

for analysis, demographic data was only available for 66 of 

those clients; this limits generalizability of the findings.  

Additionally, no demographic data about the therapists was 

collected, and it is possible that some therapist characteristics 

(e.g., years of training) could reveal interesting information.  

For example, a recent study found that particular student 

therapist variables, such as the number of client contact hours 

and days in doctoral training, were predictors of treatment 

outcome (Powell, Hunter, Beasely, & Vernberg, 2010).  

 

Future Directions 
The results of this study were mixed regarding the 

agreement between different ways for measuring outcomes.  

RC sometimes occurred when the therapist did not describe 

treatment as being successful.  In other instances, RC did not 

occur, but the therapist did describe treatment as being 

successful. It remains unclear what a clinician should do 

when these two methods produce different descriptions for 

the same outcome.  Should clinicians disregard their own 

judgment?  Or should clinicians trust their judgment and 

intuition?  The answer to these questions is beyond the scope 

of this study, but it may be important to recognize the 

discrepancies in the reports regarding therapy outcome.  To 

account for the low level of agreement between the two 

methods, it is recommended that therapists draw information 

about the disposition of treatment from a number of sources, 

such as the client’s self-report, scores on a routinely 

administered outcome measure, verbal report from a 

collateral source (e.g., a client’s spouse), clinical observation, 

and progress towards treatment goals.  Additionally, a 

Multitrait Multimethod type approach (MTMM; Campbell & 

Fiske, 1959) may be appropriate in determining whether 

treatment was successful.  Gathering information about the 

outcome of treatment from multiple sources may provide a 

more comprehensive and therefore clearer description of the 

outcome of therapy. 

One method of determining treatment outcomes that 

could incorporate both therapist perception and standardized 

measures is the use of the client’s treatment plan as the 

standard for evaluating the successfulness of treatment.  An 

appropriate treatment plan includes realistic and measurable 

goals (Leahy & Holland, 2000).  It should also include goals 

reflecting what the client expects to gain from treatment.  If 

clients meet their treatment goals, then treatment has been 

successful.  If clients do not meet their treatment goals, then 

treatment has not yet been successful.  Research has found 

that many clinicians prefer using “real world functional 

indicators” instead of standardized measures in the evaluation 

of the effectiveness of treatment (Garland et al., 2003).  Using 

the client’s treatment plan to evaluate the successfulness of 

treatment may be a valuable addition to the use of therapist 

judgment and standardized outcome measures.  

It is possible to construct treatment plans that measure 

progress based on real world functional indicators (e.g., 

improvements in school or work attendance), scores on 

outcome measures, and client reported improvement (e.g., 

subjective units of distress scales).  Research on the use of 

treatment plans to evaluate when treatment has been 

successful will demonstrate whether a client’s treatment plan 

is an appropriate method for determining the outcome of 

treatment.  It is possible that research may find the essential 

components needed for a treatment plan to be an appropriate 

means of determining when treatment is successful.  Another 

future direction for research would be to study the attitudes of 

student clinicians in regards to outcome measures and 

assessment, as it is unknown how their attitudes correspond to 

the previously studied attitudes of practicing professionals. 

 

References 

 

American Psychological Association. (2006). Evidence-based 

practice in psychology. American Psychologist, 61, 271-

285. doi:10.1037/0003-066X.61.4.271 

Anderson, E. A., & Lambert, M. J. (2001). A survival 

analysis of clinically significant change in outpatient 

psychotherapy. Journal of Clinical Psychology, 57, 875-

888. doi:10.1002/jclp.1056 

Beck, A. T., & Steer, R. A. (1990). Manual for the Beck 

Anxiety Inventory. San Antonio, TX: Psychological 

Corporation. 

Brooks, F. (2007). The living wage movement: Potential 

implications for the working poor. Families in Society, 

88, 437-442. doi: 10.1606/1044-3894.3653 

Brown, A., Williams, A., Waltman, S., & Sutton, C. (May, 

2010). Using more than intuition: Understanding client 

outcomes in a psychology training clinic. Poster session 

presented at the meeting of the Oregon Psychological 

Association, Portland, Oregon.  

Callahan, J. L., & Hynan, M. T. (2005). Models of 

psychotherapy outcome: Are they applicable in training 

clinics? Psychological Services, 2, 65-69. doi: 

10.1037/1541-1559.2.1.65 

Callahan. J. L., Swift, J. K., & Hynan, M. T. (2006). Test of 

phase model of psychotherapy in a training clinic. 

Psychological Services, 3, 129-136. doi: 10.1037/1541-

1559.3.2.129 

Campbell, D. T., & Fiske, D. W. (1959). Convergent and 

discriminant validation by the multitrait-multimethod 

matrix. Psychological Bulletin, 56, 81-105. 



WALTMAN, REX, & WILLIAMS 

 

24 

 

Cohen, J. (1988). Statistical power analysis for the behavioral 

sciences (2nd ed.). Hillsdale, NJ: Erlbaum. 

Garland, A. F., Kruse, M., & Aarons, G. A. (2003). Clinicians 

and outcome measurement: What’s the use? The Journal 

of Behavioral Health Services & Research, 30, 393-405. 

doi: 10.1007/BF02287427 

Hatchett, G. T., & Park, H. L. (2003). Comparison of four 

operational definitions of premature termination. 

Psychotherapy: Theory, Research, Training, 40, 226-

231. doi:10.1037/0033-3204.40.3.226 

Hatfield, D. R., & Ogles, B. M. (2004). The use of outcome 

measures by psychologists in clinical  practice. 

Professional Psychology: Research and Practice, 35, 

485-491. doi:10.1037/0735-7028.35.5.485 

Hunsley, J., Aubry, T. D., Verstervelt, C. M., & Vito, D. 

(1999). Comparing therapist and client perspectives on 

reasons for psychotherapy termination. Psychotherapy: 

Theory, Research, Practice, Training, 36, 380-388. doi: 

10.1037/0033-3204.36.4.380 

Jacobson, N. S., Follette, W. C., & Revenstorf, D. (1984). 

Psychotherapy outcome research: Methods for reporting 

variability and evaluating clinical significance. Behavior 

Therapy, 15, 336-352. doi: 10.1016/S0005-

7894(84)80002-7 

Jacobson, N. S., Roberts, L. J., Berns, S. B., & McGlinchey, 

J. B. (1999). Methods for defining and determining the 

clinical significance of treatment effects: Description, 

 application, and alternatives. Journal of Consulting 

and Clinical Psychology, 67, 300-307. doi:10.1037/0022-

006X.67.3.300 

Jacobson, N. S., & Truax, P. (1991). Clinical significance: A 

statistical approach to defining meaningful change in 

psychotherapy research. Journal of Consulting and 

Clinical Psychology, 59, 12-19. doi: 10.1037/0022-

006X.59.1.12 

Lambert, M. J., & Hawkins, E. J. (2004). Measuring outcome 

in professional practice: Considerations in selecting and 

using brief outcome instruments. Professional 

Psychology: Research and Practice, 35, 492-499. doi: 

10.1037/0735-7028.35.5.492 

Lambert, M. J., Morton, J. J., Hatfield, D., Harmon, C., 

Hamilton, S., Reid, R. C.,…Burlingame, G. M. (2004). 

Administration and scoring manual for the Outcome 

Questionnaire (OQ-45.2). Orem, UT: American 

Professional Credentialing Service.  

Lambert, M. J., Okiishi, J. C., Finch, A. E., & Johnson, L. D. 

(1998). Outcome assessment: From conceptualization to 

implementation. Professional Psychology: Research and 

Practice, 29, 63–70. 

Leahy, R. L., & Holland, S. J. (2000). Treatment plans and 

interventions for depression and anxiety disorders. New 

York: The Guilford Press. 

Lunnen, K. M., & Ogles, B. M. (1998). A multi-perspective, 

multi-variable evaluation of reliable change. Journal of 

Consulting and Clinical Psychology, 66, 400–410. 

Lunnen, K. M., Ogles, B. M., & Pappas, L. N. (2008). A 

multiperspective comparison of satisfaction, 

symptomatic change, perceived change, and end-point 

functioning. Professional Psychology: Research and 

Practice, 39(2), 145-152. 

Mours, J. M., Campbell, C. D., Gathercoal, K. A., & 

Peterson, M. (2009). Training in the  use of 

psychotherapy outcome assessment measures at 

psychology internship sites. Training and Education in 

Professional Psychology, 3, 169-176. doi: 

10.1037/a0016135 

Murdock, N. L., Edwards, C., & Murdock, T. B. (2010). 

Therapists’ attributions for client premature termination: 

Are they self-serving? Psychotherapy Theory, Research, 

Practice, Training, 47(2), 221-234. 

Pekarik, G., & Wolff, C. B. (1996). Relationship of 

satisfaction to symptom change, follow-up adjustment, 

and clinical significance. Professional Psychology: 

Research and Practice, 27, 202–208. 

Powell, J. L., Hunter, H. L., Beasley, L. O., & Vernberg, E. 

M. (2010). Using fine-grained indexes of therapists’ 

experience and training to predict treatment outcome in a 

university-based training clinic for children and families. 

Training and Education in Professional Psychology, 

4(2), 138-144. 

Reese, R. J., Norsworthy, L. A., & Rowlands, S. R. (2009). 

Does a continuous feedback system improve 

psychotherapy outcome? Psychotherapy Theory, 

Research, Practice, Training, 46, 418-431. doi: 

10.1037/a0017901 

Ridley, C. R. (2005). Overcoming unintentional racism in 

counseling and therapy: A practitioner’s guide to 

intentional intervention (2nd ed.). Thousand Oaks, CA: 

Sage. 

Ruscio, J. (2007). The clinician as subject: Practitioners are 

prone to the same judgment errors as  everyone else. 

In S. O. Lilienfeld & W. T. O’Donohue (Eds.), Great 

ideas of clinical science: 17 principles that every mental 

health professional should understand (pp. 29-47). New 

York: Routledge. 

Slade, M., McCrone, P., Kuipers, E., Leese, M., Cahill, S., 

Parabiaghi, A.,…Thornicroft, G. (2006). Use of 

standardized outcome measures in adult mental health 

services. British Journal of Psychiatry, 189, 330-336. 

doi: 10.1192/bjp.bp.105.015412 

Swift, J. K., Callahan, J., & Levine, J. C. (2009). Using 

clinically significant change to identify premature 

termination. Psychotherapy: Theory, Research, Training, 

46, 328-335. doi:10.1037/a0017003 

Wierzbicki, M., & Pekarik, G. (1993). A meta-analysis of 

psychotherapy dropout. Professional Psychology: 

Research and Practice, 24, 190-195. doi: 10.1037/0735-

7028.24.2.190 

Wise, E. A. (2004). Methods for analyzing psychotherapy 

outcomes: A review of clinical significance, reliable 

change, and recommendations for future directions. 

Journal of Personality Assessment, 82, 50-59. doi: 

10.1207/s15327752jpa8201_10 


