______________ Thomas Mackrill is an Assistant Professor; Frank Ebsen is a Senior Reader, and Helle Antczak is an Associate Professor at the Institute for Social Work at Metropolitan University College in Copenhagen, Denmark . Copyright © 2015 Advances in Social Work Vol. 16 No. 1 (Spring 2015), 67-77 Developing Scales for Smartphone Applications Together: Youth and Municipal Case Worker Perspectives Thomas Mackrill Frank Ebsen Helle Antczak Abstract: This article reports the initial findings of a Danish action research project aiming to develop a digital tool that young persons can use to inform their municipal case workers about their well-being. The project vision is an integrated system with a smartphone interface for youth clients and a web interface for case workers, whereby both parties can track how the young person is doing. Three meetings were held between researchers, software developers, young persons, and their case workers. The young persons rejected self-monitoring on a normative scale. They rejected a scale proposed by case workers that encouraged them to focus on a positive future, favoring a scale that enabled them to focus on their well-being being low. The young persons and case workers disagreed about how data regarding change should be presented. Case workers preferred a graph that highlighted risk, whereas young persons favored a graph that emphasized positive change. Key words: Social work, apps, innovation, scales, youth, action research. This pilot study addresses an important field – the use of outcome measures as collaborative tools in social work. In recent years there has been an increased focus on implementing routine outcome measurement into counseling practice with the aim of improving effectiveness (Lambert, 2010). Much of the literature on this has concerned the psychometric properties of the various scales developed and significance of using these methods for outcomes. A key finding has been that the use of routine measurement reduces drop out in routine practice (Lambert, 2010). Some developers of measures note that routine outcome measurement is not just a matter of measurement and that such tools also aim “to give youth a voice in the services they receive” (Duncan, Sparks, Miller, Bohanske, & Claud, 2006, p. 72). The tools are seen as ways clients can communicate about their experiences of progress with their counselors. They can be viewed as part of collaboration about an intervention. Securing the incorporation of the clients’ voice by means of a standardized tool may be central to why the use of such tools can reduce drop out. The steering committee of the American Psychological Association Division 29 task force identified goal consensus and collaboration between client and counselor as a “demonstrably effective” element of the therapeutic relationship (Steering Committee, 2002). They based this conclusion on a review of studies of the goal consensus and collaboration (Tryon & Winograd, 2002) and on research into the working alliance based on Bordin’s (1979, 1994) pan-theoretical concept. Central elements of Bordin’s concept are agreement about counseling goals and agreement about the tasks occurring during counseling, with the third and final element being the bond between the client and the counselor. As well as using routine outcome measurement to enhance outcomes, other brief collaborative tools have been developed. The most widely researched of these is the Session Rating Scale (Duncan et al., 2006), a four item measure based on Bordin’s Mackrill et al./DEVELOPING SCALES FOR APPS 68 (1979) concept of the working alliance. The Session Rating Scale is generally used alongside a brief outcome measure called the Outcome Rating Scale (Miller, Duncan, Brown, Sparks, & Claud, 2003). Another noteworthy collaborative tool is the Therapy Personalisation Form (Bowen & Cooper, 2012), where the client, prior to counseling, reports how they would like their counseling in relation to a wide range of parameters. When self-report scales are used alongside interventions, they are not mere research or evaluation tools; they are an intrinsic part of an intervention (Mackrill, 2008, 2011). There are very few studies of clients’ experiences of using routine outcome measurement. Anker, Sparks, Duncan, Owen, and Stapnes (2011) asked a group of clients (n=377) to rate their experience of using the Outcome and Session Rating Scales as either “disturbing,” “not important,” “useful/helpful,” or “do not remember.” Of these clients, 60.7% reported the scales were “useful/helpful,” 30% noted “not important” and 6.6% experienced the scales as “disturbing.” In a similar quantitative study of 300 clients using the CORE tracking system (Barkham et al., 2010), 47% were “quite happy,” 45% “didn’t mind,” 3% were “not keen” and 0.8% “disliked it” (Unsworth, 2009). In a qualitative study of family therapy, Sundet (2011) noted that families generally found the Outcome Rating Scale a feasible tool for conversation that gave a good image of how different family members were doing and offered an opening for conversations about progress. Sundet (2011) also noted that one family rejected the use of the scale as they would rather give feedback verbally and some clients found the subscales on the Outcome Rating Scale too broad and found it difficult to point to where they were on the scales. In a further qualitative study of individual psychotherapy with six clients where the Outcome and Session Rating Scales were used, Pedersen (2012) found differences among clients, noting: some clients experienced the Outcome Rating Scale (ORS) as a great therapeutic tool to think through their situation and decide what to focus on. They liked how the ORS structured the opening of the session and saw the ORS as a good way to quickly give the therapist a lot of information about their condition and situation. Other clients scored the ORS without attaching much meaning to it, but some clients experienced the ORS as distracting. It undermined the presence and removed focus from more important issues. (p. 25) Pedersen (2012) also noted that clients found expressing a variety of experiences on the subscales of the Outcome Rating Scale difficult, and that while many found tracking their progress using a scale across sessions meaningful, some did not feel this approach to evaluation was meaningful and in line with their therapeutic goals. In a qualitative study using brief CORE outcome measures, Unsworth (2009) found that clients generally felt that a visual representation of their condition and progress was helpful and they were positive about using the scales in dialogues with their counselors, although some clients found it hard to complete measures if they were distressed. The above studies all focus on client experiences of outcome measurement in routine counseling practice. To our knowledge, no studies have been conducted of young persons’ experiences of routine self-report tools in the context of collaborative tool development. Furthermore, the above studies all focus on the use of outcome measurement in clinical settings. The context of the present pilot study is the ADVANCES IN SOCIAL WORK, Spring 2015, 16(1) 69 development of a routine outcome measure for use in municipal case management settings, which is also new territory. This article reports differences in young person and case worker perspectives on the design of a single item self-report well-being scale for communication purposes. The data are from the first cycle of a Danish action research project aiming to improve communication between young people and their municipal case workers from social services. The action research project is a collaboration between researchers, software developers, and young persons and their municipal case workers. It aims to develop a digital communicative tool that young persons can use to inform their case workers about their well-being as well as about specific, agreed- upon behaviors and activities. The project vision is an integrated system with a smartphone interface for young persons and a web interface for case workers. The project is inspired by psychotherapy research into practice-based evidence where a range of similar systems have been built for use in psychotherapy (see for example Barkham et al., 2010; Lambert, 2010; Miller et al., 2003). The project is also inspired by a local Copenhagen project, “MinVej,” where a recovery-supporting smartphone application has been developed not just for, but with psychiatry users. The challenge in this action research project is to develop a system that serves both the interests of young social service users and their caseworkers. As well as providing caseworkers with feedback about their young clients’ life, the app interface also aims to offer the young person the opportunity of tracking their own change for personal development purposes. Developers of self-report scales are generally interested in whether scales are reliable, that is to say consistent in their ability to measure, and valid, referring to whether they measure what they intend to measure. Instead, this pilot study focuses on whether young persons who use a tool find it meaningful and appropriate. This issue is particularly important if a self-report scale is to be used routinely by a young person, not in the immediate vicinity of a professional prompting them to self-report. This is a key way in which the tool developed in this project differs from those used in routine outcome measurement in clinical settings. In this project, action researchers, software developers, caseworkers, and young persons were a participatory community of inquiry engaging in the practical problem of developing a tool to enhance communication (Reason & Bradbury-Huang, 2008). Rather than addressing client and professional experiences of using an outcome measure, this study addresses client and professional participation in developing an outcome measure. Method Six caseworkers, who were all qualified social workers with bachelor degrees in social work, were recruited from three municipalities in Denmark, with two social workers from each municipality, five women and one man. All the social workers worked with youth clients in a family or youth social service setting. The social workers recruited six young persons whom they had contact with to participate in the project: three young women aged 14, 16, 17 and three young men aged 16, 17, and 18. The young persons were recruited primarily because their social workers judged they had the resources and willingness to participate. They were not recruited because they had a particular type of problem. The researcher leading the project had a background in counseling research. Mackrill et al./DEVELOPING SCALES FOR APPS 70 Three group meetings were held where the participants discussed a design presented to them as a Microsoft Powerpoint© presentation. The counseling researcher led the first two meetings, and a second researcher took detailed field notes where the perspectives of the participants, the topics of discussion, the atmosphere in the meeting, and the decisions made were noted. The final meeting was led by the software developers. The young persons were asked to give their opinions about the design but also to consider what other young people, who they knew, might think about the design. The social workers were similarly asked to consider what their colleagues might think about the design, as well as themselves. After each meeting, the design was revised based on the input from participants, and a new version was presented at the following meeting. The meetings were divided into two stages. First, social workers provided feedback on the design without the young persons being present. Then, the young persons provided feedback on the design with half of the social worker group present. The social workers were instructed to be silent when the young persons gave their feedback so the young persons could voice their opinions without interference. Towards the end of the meetings, there was dialogue between the social workers and young persons. The meetings were generally lively, and all participants voiced their views about the designs. The counseling researcher facilitated the process so all voices were heard. There was a general sense of excitement about having the opportunity of developing the system. The young persons were offered food and drink during the meetings, and they were given cinema vouchers for two, including a drink and popcorn, as a gift for participating in the project. The findings presented in this paper were fed back to the participants during later meetings to check their validity (Reason & Rowan, 1981), and the participants confirmed that we had understood their perspectives correctly. The analysis thus was conducted by reflecting on discrepancies between youth and social worker perspectives, by examining the contents of the field notes after the meetings, and by playing perceived differences in perspectives back to the participants in subsequent meetings to develop, confirm, or challenge the analyses. In Denmark, there are no ethics committees that approve social science research such as this. There are only ethics committees in the field of medicine. Young persons participated voluntarily, and parental consent was obtained for young persons under the age of eighteen. Some young persons who were asked to participate declined without this affecting their access to social support in any way. All data were secured according to the regulations of the Danish Data Protection Agency. The identities of the social workers and young persons are concealed. Results The young clients rejected the normative scale Figure 1 shows the initial draft version of the well-being scale. Color-coding with red, orange, and green sections was proposed to enhance the intuitive communicative aspect of tool, drawing on the image of a traffic light where red signals stop, orange signals that a stop may be imminent, and green signals go. The counseling researcher proposed that the green section should take up most of the scale, arguing that a satisfactory overall experience of well-being during the week, which is neither “good” nor “bad,” should be in the middle of the scale. The counseling researcher argued that this was important to avoid giving the young persons an unrealistic ADVANCES IN SOCIAL WORK, Spring 2015, 16(1) 71 expectation about how good a week could be expected to be. According to the counseling researcher, a week with well-being in the middle of the scale was a sign of success. The young people all rejected this view. They said, “We know we can’t expect to have a fantastic time all the time,” “We know our lives aren’t as good as other people’s lives,” and “We don’t need to be reminded about it by a scale.” They wanted a scale where they had a chance of progressing from one side of the scale to the other, rather than a normative scale where they might just make it to the middle. Figure 1. Well-being - first version. Figure 2. Well-being – case worker version. Figure 3. Well-being - young person version. The young clients rejected a strengths focus The social worker group also disagreed with the initial design, but for different reasons. In their view, the green section had the right length, but they argued that it should be on the left side of the scale rather than on the right. See Figure 2. They were unanimous in this view. They argued that they had been trained in focusing on client strengths. When the red section was on the left, this was what first caught their eye, as they read from left to right. They wanted their clients first to focus on the green section as a way of drawing their clients’ attention to the potential for improved weekly well-being, that there was hope for a better future, and that they could have better life. The young clients rejected the social workers’ design unanimously, stating that their well-being was generally dissatisfactory, and that a design that intuitively made them focus on how good a week could be was provocative and demotivating, as such a life was such a long way off. They wanted their social workers to appreciate how difficult their lives were. In collaboration with the young persons, we developed the scale that can be seen in Figure 3. Mackrill et al./DEVELOPING SCALES FOR APPS 72 A positive change versus a risk reduction focus The social worker group and the young person group also disagreed about how past data should be viewed. While they agreed that a graph might be an appropriate way of viewing changes in well-being over time, they disagreed fundamentally about how data should be presented. The social workers favored the red zone being up and the green zone being down on the y axis. Figure 4 shows this, with a positive change as a curve sloping down from left to right. The social workers preferred this view as they felt it represented a change in the level of risk. For them, “low” well-being was a “high” risk, and it should therefore be positioned at the top of the viewing. A “high” level of well-being was a “low” risk that should be positioned at the bottom of the graph. This was intuitively most correct from their perspective. This was the opposite of the initial draft. The social workers' version of the graph with red at the top and green at the bottom was then presented to the young persons who unanimously dismissed this graph. They wanted the red section representing “low” well-being to be at the bottom of the y axis. They wanted to be able to record “positive change,” which for them intuitively meant movement in an “upwards” direction from left to right. See Figure 5. The young people focused on increases in well-being, while the social workers focused on reducing risk. Figure 4. Well-being graph – case worker version. ADVANCES IN SOCIAL WORK, Spring 2015, 16(1) 73 Figure 5. Well-being graph - young person version. Discussion The young person and social worker groups were surprisingly unanimous in their viewpoints. At a glance, this may seem to suggest that the results presented were the result of groupthink (Janis, 1982). However, this is not the case as both the young persons and social workers voiced a range of views at the meetings, and there were frank exchanges of views in each group. The counseling researcher made sure there was space to reflect on one’s own perspective during meetings, and participants also had the opportunity to reflect on their views between meetings. The counseling researcher supported and repeated the many different perspectives that were expressed, so they could be discussed. It was only after a discussion of the many viewpoints that the groups reached a consensus. The counseling researcher was not aiming for consensus in any way and questioned the unanimity in the groups’ perspectives. The counseling researcher was actually surprised by the extent of the participants’ agreement, particularly as the groups were not characterized by dominant members imposing their views on others. The young clients had very clear opinions about how the self-report scale should be designed. Thus, this study highlights the importance of user involvement in scale development. This is particularly important when dealing with a vulnerable group whose perspectives are often ignored. When using self-report scales for routine assessment, it is important to follow advice about using an established instrument rather than one with less scientific credibility (Paulhus & Vazire, 2007), but at the same time it is also important to check whether users have been involved in scale Mackrill et al./DEVELOPING SCALES FOR APPS 74 development or whether their experiences of using the scale routinely have been studied. There may be a difference between asking clients about their experiences of using a scale after a scale has been developed and asking them their experiences during the development of a scale, when they know their views will be taken into consideration in the development of the scale. The young person’s rejection of the normative scale is significant. When using scales in routine self-assessment with vulnerable groups, the young person needs to be able to see their own personal development in relation to the scale as a whole. Using norm-based scales for groups who have trouble living up to the norms inherent in the scales is a poor choice, as the group will consistently be reminded of their failure to live up to the norms. The difference between the social workers’ wish to have a scale that drew attention to the young persons’ strengths and possibilities of attaining a positive future and the young persons' wish to have a scale that reflected that their lives were difficult was a surprise to the social workers. The social workers argued that this was a perspective they had been trained in. The key assumption in a strengths perspective is that “every human being has, within or around, resources, capacities, and assets that can be mobilized to overcome adversity or to inspire a better quality of life” (Saleebey, 2008). Most of the social workers who took part in the study were trained in this perspective at the Institute where the researchers were employed. Thus this finding reflected back on the educational practices of the researchers’ place of work. The strengths perspective is a reaction to social work that was overly focused on client problems rather than client resources. Striking a balance between recognizing client problems and maintaining hope for a more positive future is a constant challenge in social work. This study showed that the social workers’ strengths perspective was out of synch with their clients’ perspectives. A discrepancy in perspectives, where social workers maintain hope for clients who have little or no hope for themselves, is an intrinsic part of social work. However, if social workers are so hopeful that clients do not feel their social workers appreciate or understand the problems they face, the strengths perspective becomes a hindrance and a problem in itself. A strengths perspective can perhaps get out of hand if not appropriately combined with a problem focus. The discrepancy between the young clients’ focus on positive changes in well- being and the social workers’ focus on risk reduction is not surprising. Risk reduction is a primary goal for municipal case workers, which reflects the legal foundation on child protection that underlies their work. The young persons do not reflect on the law in the same way as the social workers but have a more general interest in change for the better. In their book, Metaphors We Live By, Lakoff and Johnson (1980) argue that metaphors shape our perceptions. One metaphor they point to is the metaphor of direction, suggesting that we generally perceive “good” and “more” as “up,” and “bad” and “less” as “down.” In some ways, this corresponds with the findings presented in this study. On the graph, the young persons intuitively want “more” well-being to be “up,” and the social workers similarly want “more” risk to be “up.” However, high risk is not “good,” and the social workers agreed that high risk should be “up.” This suggests that “more” may take precedence over “good” when intuitively determining what is “up.” ADVANCES IN SOCIAL WORK, Spring 2015, 16(1) 75 We used the above analysis in the further development of the tool. We changed the well-being scale so it was in line with young persons’ wishes as seen in Figure 3. We also developed the graph in accordance with the young persons’ wishes as seen in Figure 5. We discussed the possibility of having a positive change focused graph (Figure 5) on the young person’s smartphone interface and a risk-oriented graph (Figure 4) on the case workers’ web interface. We choose to have a positive change focus (Figure 5) on both interfaces, as we wanted the social workers and young persons to have the same view, so the web interface could be easily used for talks about how things were going at meetings between the young persons and their case workers, without having to explain why the graphs looked different. This pilot study is obviously limited by its small sample size and the fact that participants were all willing participants. Some young people in contact with municipal social services would not want to participate in such a project, and we cannot be sure that participants who are willing to participate can represent the perspectives of the unwilling, even though the young persons were asked to consider what other young people, who they knew, would think. This pilot study is a first step in tool development. We plan to run a test-retest of the single item scale and test it against other scales such as the Outcome Rating Scale (Miller et al., 2003) and the Young Person CORE (Twigg et al., 2009). This would enable the scale to be used both for monitoring change and for collaboration. Further studies of youth and social worker perspectives will take place in connection with testing a trial version of the scale in the second cycle of this action research project. References Anker, M., Sparks, J., Duncan, B., Owens, J., & Stapnes, A. (2011). Footprints of couple therapy: Client reflections at follow up using a mixed method design in routine care. Journal of Family Psychotherapy, 22, 22-45. Barkham, M., Mellor-Clark, J., Connell, J., Evans, C., Evans, R., & Margison, F. T. (2010). Clinical Outcomes in Routine Evaluation (CORE) – The CORE measures and system: Measuring, monitoring and managing quality evaluation in psychological therapies. In M. Barkham, G. E. Hardy, & J. Mellor-Clark (Eds.), Developing and delivering practice-based evidence: A guide for the psychological therapies (pp. 175-220). Chichester, England: Wiley. Bordin, E. S. (1979). The generalizability of the psychoanalytic concept of the working alliance. Psychotherapy: Theory, Research and Practice, 16, 252-260. Bordin, E. S. (1994). Theory and research on the therapeutic working alliance: New directions. In A. O. Horvath & L. S. Greenberg (Eds.), The working alliance (pp. 13-37). New York: John Wiley. Bowen, M., & Cooper, M. (2012). Development of a client feedback tool: A qualitative study of therapists’ experiences of using the Therapy Personalisation Forms. European Journal of Psychotherapy & Counselling, 14, 47-62. Duncan, B., Sparks, J., Miller, S., Bohanske, R., & Claud, D. (2006). Giving youth a voice: A preliminary study of the reliability and validity of a brief outcome measure for children, adolescents, and caretakers. Journal of Brief Therapy, 5, 71-87. Mackrill et al./DEVELOPING SCALES FOR APPS 76 Janis, I. L. (1982). Groupthink: Psychological studies of policy decisions and fiascoes. Boston: Houghton Mifflin. Lakoff, G., & Johnson, M. (1980). Metaphors we live by. Chicago: The University of Chicago Press. Lambert, M. J. (2010). Prevention of treatment failure. The use of measuring, monitoring, and feedback in clinical practice. Washington DC: American Psychological Association. Mackrill, T. (2008). Solicited diary studies of psychotherapeutic practice - pros and cons. European Journal of Psychotherapy & Counselling, 10, 5-18. Mackrill, T. (2011). Diaries. In L. L’Abaté & L. Sweeney (Eds.), Research on writing approaches in mental health (pp. 55-65). Bingley, United Kingdom: Emerald Group Publishing Limited. Miller, S. D., Duncan, B. L., Brown, J., Sparks, J., & Claud, D. A. (2003). The Outcome Rating Scale: A preliminary study of reliability, validity, and feasibility of a brief visual analog measure. Journal of Brief Therapy, 2, 91-100. Paulhus, D. L., & Vazire, S. (2007). The self-report method. In R. W. Robins, R. C. Fraley, & R. F. Krueger (Eds.), Handbook of research methods in personality psychology (pp. 224-239). New York: Guilford. Pedersen, M. J. (2012). Clients' and therapists' experience of tracking (Doctoral thesis, Institute for Psychology, Copenhagen University). Available from http://www.e17.dk/medier/628889 Reason, P., & Bradbury-Huang, H. (Eds.). (2008). Introduction. In Sage handbook of action research: Participative inquiry and practice (pp. 1-10). London: Sage. Reason, P., & Rowan, J. (Eds.). (1981). Issues of validity in new paradigm research. In Human inquiry: A sourcebook of new paradigm research (pp. 239–250). New York: John Wiley and Sons. Saleebey, D. (2008). The strengths perspective: Putting possibility and hope to work in our practice. In B. W. White (Ed.), Comprehensive handbook of social work and social welfare (vol. 1, pp. 132-142). Hoboken, NJ: John Wiley & Sons. Steering Committee. (2002). Empirically supported therapy relationships: Conclusions and recommendations of the Division 29 Task Force. In J. C. Norcross (Ed.), Psychotherapy relationships that work: Therapist contributions and responsiveness to patients (pp. 441-443). New York: Oxford University Press. Sundet, R. (2011). Collaboration: Family and therapist perspectives of helpful therapy. Journal of Marital and Family Therapy, 37, 236-249. Tryon, G. S., & Winograd, G. (2002). Goal consensus and collaboration. In J. C. Norcross (Ed.), Psychotherapy relationships that work: Therapist contributions and responsiveness to patients (pp. 109-125). Oxford, UK: Oxford University Press. ADVANCES IN SOCIAL WORK, Spring 2015, 16(1) 77 Twigg, E., Barkham, M., Bweick, B. E., Mulhern, B., Connell, J., & Cooper, M. (2009). The Young Person’s CORE: Development of a brief outcome measure for young people. Counselling and Psychotherapy Research, 9, 160-168. Unsworth, G. (2009). Therapists’ and clients’ perceptions of using CORE-Net and ARM-5 in the NHS (Doctoral Thesis, Faculty of Health and Medical Sciences, Division of Health and Social Care, University of Surry, UK). Available from http://ethos.bl.uk/OrderDetails.do?uin=uk.bl.ethos.510359 Author note Correspondence concerning this article should be addressed to Thomas Mackrill, Metropolitan University College, Institute for Social Work, Kronprinsesse Sofies Vej 35, 2000 Frederiksberg, Denmark. thma@phmetropol.dk. Ph. +45 72489341