Applying Ethical Theory to the Evaluation of a Community Organization 1 Applying Ethical Theory to the Evaluation of a Community Organization Rebecca Wulforst, Giulia Molisani, Ryan Barney, Colleen Crawford, Alexandra Kwiek, Daylinis Lugo, Cody Niver, Jonathon Stowell, and Ashley Villone Nazareth College of Rochester Executive Summary This report will take an in-depth look at a community organization’s programs through the analysis of the data collected from activity observations using the Out of School Time (OST) observation instrument. The Out-of-School Time Observation Tool was developed by Policy Studies Associates, Inc. with the support of the Charles Stewart Mott Foundation (Pechman et al. 2008). This report summarizes the research process, presents the main findings, and suggests how the community organization can improve their programs. Furthermore, we explain how this service learning experience is explicitly connected to the norms of caring and utilitarianism. Lastly, there is a learning reflection which discusses our collective learning based on team processes, ethical content, and organizational practices at the community organization. Research showed that staff was rated low in the categories of asking youth to expand upon their ideas and thoughts and employing varied teaching strategies. Based on these findings, we suggest that if staff make the necessary changes to their teaching methods, youth will become more engaged and have a more positive learning experience within their activities. Nevertheless, staff was rated high in the categories related to relationship building. The strong relationships formed between the staff and youth were apparent throughout our observations. Research Questions Our group was given the task to reflect on what we took away from our observations, connect these reflections to ethical theories, and to make suggestions for future process improvements. We formulated research questions to coincide with this task. We questioned what aspects of the community organization’s programs could be improved and how does the evaluation of these community programs relate to ethical theory. Methodology This research was conducted in the Spring of 2012 by a group of nine individuals. The community organization we partnered with offers programs and other forms of assistance to community members. Specifically, we observed after school programs offered to youth. Pre-work In order to learn more about the organization, we visited the community based organization in order to introduce ourselves to the environment and diverse programs that are offered. Once we were exposed to the culture and operations, select group members met with the manager of data, research, and program evaluation to establish the foundation of the service learning project. During this meeting, group members discussed the different programs that the organization offers youth and how our observations will be of assistance to their programs. We also discussed possible training times and exchanged e-mails to better improve communication. 2 Training A two hour training session was conducted by the organization’s staff in which all team members received instructions on how to utilize the observation instrument. In order to provide reliability to our research, a video was shown that provided an example of what we would be observing. While watching the video, each team member rated the specific categories exemplified on the observation tool. The Out of School Time (OST) indicator rating guide consists of a variety of different categories. These categories include relationship building and participation for youth as well as relationship building and instructional strategies for staff. Also included in the guide are categories for the content and the structure of each individual activity, their environmental context, and the observer’s synthesis and rating. Below is the rating scale used to score the different categories. Five (5.0) is the standard rating for grading criteria, as stated by the OST guidelines (Pechman et al. 2008). -----1----- -----2----- -----3----- -----4----- -----5----- -----6----- -----7----- Exemplar is Exemplar is Exemplar is Exemplar is highly not evident rarely evident evident/implicit evident and consistent Observations In order to collect and compile the data necessary to complete the project, the group was divided up into teams for each specific after school program: Program One and Program Two. These programs invite youth to play games and participate in educational activities. The OST guidelines stated that each team must observe their assigned activities for a maximum of 15 minutes and fill out the observation instrument based on what they observed. Once all observations were completed, the group met as a whole and compiled all data into an Excel spreadsheet. This sheet calculated the average ratings for each program and also for the individual categories for youth and staff. Data Analysis Program One What the Staff Does Well When observing the staff interacting with the youth, it was clear that they showed positive affect towards youth and communicated goals, purposes, and expectations clearly. On average, each of these categories received an OST Indicator rating of six (6.0). Overall, in all the activities observed, staff seemed to be friendly and positive when interacting with the youth. Furthermore, in regard to communicating goals, in the gym class the staff explained the purpose of the game capture the flag to the youth before beginning to play. The youth understood that they needed to reach the other side of the gym without being tagged by the opposite team and retrieve the flag. 3 Areas That Need More Attention Overall, observations demonstrated that the youth did not meet the standards in the categories of: listen actively and attentively to peers and staff (3.8 average), contribute opinions, ideas, and/or concerns to discussions (3.5 average), and take leadership responsibility/roles (3.0 average). It was observed that the youth were most interested in their personal conversations than with group discussions about the activity occurring. For example, in the Promoting Alternative Thinking Strategies (PATHS) activity, some youth did not actively listen to the book that was read by the staff member. Instead, they were designing Easter eggs and were not concerned with the current discussion. In the observation, only one youth was actively engaged and shared his ideas about the book. Observations demonstrated that staff members can improve mainly in their instructional strategies. The categories that were rated the lowest were: ask youth to expand upon their answers and ideas (2.5 average), challenge youth to move beyond their current level of competency (3.0 average), and plan for/ask youth to work together (3.5 average). For the Science, Technology, Engineering, & Math (STEM) activity, the youth were asked to create a silly putty material with water and baking soda. The youth completed this task, but staff members never asked them to evaluate “why” or “how” this material was made. Furthermore, in the gym activity staff did not give students the opportunity to discuss strategy, rather they told them the best way to capture the flag. In both examples, there is a lack of questioning in youth’s ideas. Program Two What the Staff Does Well During our observation there were many areas in which the staff met or exceeded program expectations. The staff was very personable and engaged in conversation with the kids on subjects that were not directly related to the actual program. The OST indicator for engaging personally with youth had an average of 5.0, and the OST for showing positive affect toward youth had an average of 5.5. Overall there were never any signs of favoritism by the staff and they were consistently positive, patient, and tolerant towards the youth. The staff helped the groups or individuals, but they were never controlling of the situation. Our observations showed that activities were geared towards communication, team building, and encouraging positive relationships/interactions in teaching interpersonal skills. Areas That Need More Attention Overall, observations showed that the youth did not meet the standards in the categories of: listen actively and attentively to peers and staff (3.5 average), contribute opinions, ideas, and/or concerns to discussions (4.0 average), and respect one another (4.0 average). During the STEM activity, although the staff initially tried to provide instruction and guidance, the youth disregarded their instruction. Rather than paying attention, they were engaged in conversations that were unrelated to the subject at hand. Furthermore, there were several instances of disrespect between youth. For example, one specific youth consistently acted inappropriately towards other youth by saying, “I will punch you in the face,” in a joking manner. 4 Overall, observations demonstrated that the staff did not meet the standards in the categories of: encourage the participation of all (3.5 average), ask youth to expand upon their answers and ideas (2.5 average), and challenge youth to move beyond their current level of competency (3.0 average). There was one youth who never participated in any of the activities and was generally quiet around the other children. This specific youth remained isolated during the indoor basketball game. Not once was she encouraged to participate. In relation to this example, the usage of cell phones in the classroom distracted from learning and made it difficult for staff to challenge the youth and ask them to expand their ideas. Ethical Analysis The analysis of data collected at the organization’s various after-school programs provided a connection with ethical theories studied through Management Ethics offered at Nazareth College. Specifically, we chose to analyze the norms of caring and utilitarianism that were introduced in Gerard Cavanagh’s (2010) American Business Values a Global Perspective. Norm of Caring According to Cavanagh, the norm of caring can be defined as, “responsibility to a person because of relationship” (2010, 88). The relationship of care is evident in the connections established between the youth and staff and the organization and community. The responsibility of the youth and staff was apparent in our observations. In an activity involving reading chapters from a book, a staff member engaged youth to think about what means the most to them in their lives. It was clear that the staff member’s goal was to provide guidance to the youth. Likewise, it was the responsibility of the youth to participate in the class activity. Furthermore, the relationship between the organization and community was observed through their numerous services. The organization offers social, developmental, and educational programs aimed at building trust, communication, and responsibility with a strong emphasis on care. Cavanagh also maintains that, “Caring engages our emotions. In order to incorporate the interests of others into one’s decision-making processes, one must be able to feel and to empathize with those that are affected by one’s decisions” (2010, 94). Given our various examples, it was obvious that staff, youth, and the organization as a whole exemplified high levels of empathy and genuineness. Alternatively, it can be argued that the staff and the youth are not following the norm of caring. By scoring low in the areas of listen actively and attentively to peers and staff and respect one another, it is apparent that youth are not valuing their relationships appropriately. Similarly, staff scored low in the areas of encourage the participation of all, ask youth to expand upon their answers and ideas, and challenge youth to move beyond their current level of competency. As mentors, the staff at the community organization should help the youth expand their intellectual capacity. 5 Norm of Utilitarianism The utilitarian norm says that, “the ethical action is that which produces the greatest good for the greatest number of people” (Cavanagh 2010, 92). The staff of the organization demonstrates utilitarianism through their efforts to decrease the number of youths on the streets between the hours of 3:00 p.m. and 6:00 p.m. Discussion with organization staff informed us that this is when crime is at its highest (January 2012). By providing a structured and educational environment, youth have a safe place to go after school and crime can potentially be reduced, benefiting the community. These programs are geared towards improving youths’ communication, interpersonal, and relationship building skills that empower them to behave respectfully towards others and be more aware of the consequences of their actions. These skills will prove beneficial in the future when youth are advancing to adulthood and entering the working world. Thus, these skills can be applied to all aspects of life. The findings show that the organization is effectively teaching the skills necessary to build a strong community. These skills are transferred from the individual, to the family, and finally to the community. Suggestions for Improvement Our data analysis indicated that some improvement is needed. Overall, we found that youth scored low in the category of contributing opinions, ideas and/or concerns to discussion. Furthermore, the staff did not challenge youth to move beyond their current competency level because they did not ask questions such as “why”, “who” and “if”. This may be a solution to the issue regarding youth’s low level of contribution of their opinions and ideas because they need to be prompted by the staff to share openly. In addition, staff could implement teamwork within the gym activity by asking youth to develop strategies together. If the youth were given this opportunity, they could work cooperatively to capture the flag. Seeing as it can be argued that the youth and staff were not respecting the norm of caring, the community organization needs to create a culture that promotes ethical behavior. There are a variety of ways in which they can do this. According to Cavanagh, “difficulties in arriving at an ethical judgment often stem from our lack of knowledge of ethical norms” (2010, 112). Therefore, the community organization needs to educate both the youth and the staff. Staff could undertake the challenge of learning about ethical norms and concepts, and could transfer their learning directly to the youth through various activities. Another important aspect in creating a culture that promotes ethical behavior is in promoting the values of the firm. The community organization has to maintain a strong commitment to ethical behavior, which is important considering work climates have a strong influence on the actions of members within an organization (Cavanagh 2010, 220-221). All of the staff and departments at the organization should aspire to be good models for the youth. Therefore, the values that the organization introduces in their mission statement should be publicly communicated and practiced. 6 Learning Reflection Completing our service learning project provided us with skills and knowledge that we will use throughout our future careers. We discovered that data collection can be a confusing process, considering there are a number of steps to reach the final objective. Predetermined goals and expectations, with regard to data collection, were key to being successful. Having the OST tool made the observation process more efficient because it showed clear guidelines as to what criteria the organization would like their programs to meet. After completing our observations, we consider the OST indicator to be both reliable and valid because there was a consistency in the results found and the instrument ultimately measured what it was supposed to measure. Overall, our assessments were very similar when measuring one program at the same time. This shows that the different personalities of the raters will not have a large effect on the findings, and therefore the information will remain reliable. Furthermore, we were able to collect data about the quality of the programs the organization offers by rating certain components of the activities such as relationships and youth participation. This is precisely what the OST indicator is intended to measure (Yohalme et al. 2009). The OST observation tool is a necessary ingredient in the plea for government assistance and private donations that will provide assistance to help the organization continue to offer the numerous outreach programs aimed at strengthening families and building a community. These tools are used to constitute the benefits, modifications, and opportunities being achieved by the organization, proving that the programs are making substantial changes within the community and are worthy of funding assistance. Obtaining government and private funding poses a tension for the organization and constitutes a direct rival of ethical norms and values. The tension is present between the staff members’ desire to help those in the community and the struggle to continue to obtain government and private funding year after year (Personal Communication, Organization Staff, January 2012). The project presented us with both positive and negative learning. For example, we can take away the optimistic values, such as community involvement and building relationships, which are represented at the community organization, and use them in our future managerial positions. This is especially important because managers have the potential to influence every individual or stakeholder in a corporation. Furthermore, it prepared us for active civic participation in a diverse, democratic society because, while observing, we were in the vicinity of people of many diverse backgrounds. In addition, it gave us the opportunity to gain practical experience in a potential career field. Finally, we have learned the best practices of collecting and analyzing data, which will be useful in our organizations to understand the impact of our managerial decisions. There were many successes and obstacles that we experienced through this service learning project. First and foremost, we succeeded when we used email and Google docs to communicate digitally, allowing group members to work on the paper at any point in time. Also, training gave us an example of what exactly we would be experiencing when completing the observations. Furthermore, dividing the observations into programs was extremely helpful especially when carpooling to the organization. There were a variety of different obstacles faced when completing this service learning project. One of these obstacles was synthesizing the information. With such a variety of data, it made it challenging to distinguish what was most relevant to our research. Secondly, we felt that 7 observing for 15 minutes was an inadequate amount of time to thoroughly analyze the activities within the programs. Conclusion This service learning project with the community organization has increased our knowledge of the assistance offered to the community, and its youth allowed us to apply our knowledge of ethical theory to the community organization’s efforts. Our observations and suggestions were shared with the community organization leaders through a formal presentation and a paper, which gave them suggestions for improvement and general feedback. As stated previously, proof that the organization is continually looking to improve its programs is how it continues to receive funding through government grants. The community organization leaders appreciated our feedback and took into consideration our suggestion for more training to help staff learn different teaching strategies and engage youth at their highest level of competency. This service learning project was a prime opportunity to build relationships with community members and apply our ethical knowledge to a real-world situation. Our experience working with the community organization and the chance to use the different theories we’ve learned has engrained fundamental analysis, collaboration, and problem-solving skills that we will use in our future careers when faced with an ethical dilemma. The theories that we applied in the community organization have a broad range of use. For example, in the field of business, management may need to analyze the effects of a layoff within the company. When applying utilitarianism, management may ask if it is the best choice for the entire group. Using the norm of caring, management can think about whether their decision enhances or maintains their relationship with the person affected by that decision. We believe that the education and use of these theories can help promote a more ethical and just workplace. We would like to express our sincerest appreciation to Dr. Jennifer Leigh of Nazareth College for her willingness to assist us in our efforts and her constant words of advice and encouragement. References Cavanagh, Gerard F. 2010. American Business Values A Global Perspective. 6th edition. Upper Saddle River, NJ: Prentice Hall. Yohalem, Nicole, Alicia Wilson-Ahlstrom, Sean Fischer, and Marybeth Shinn. 2009. Measuring Youth Program Quality: A Guide to Assessment Tools. 2nd edition. Washington, D.C.: The Forum for Youth Investment. http://www.forumfyi.org/files/MeasuringYouth ProgramQuality_2ndEd.pdf. Pechman, Ellen M,, Monica Mielke, Christina Russell, Richard White, and North Cooc. 2008. “OST Observation Instrument and Report on Its Reliability and Validity.” Policy Studies Associates. http://www.policystudies.com/studies/?id=30. 8 9 Category Activity: PATHS STEM Gym Art Club AVERAGE: Minute STEM ScavengerBasketballAVERAGE: Relationship Building: Youth Are friendly and relaxed with one another. 3 5 6 7 5.3 5 3 5 6 4.8 Respect one another. 5 7 4 7 5.8 4 3 6 3 4.0 Show positive affect to staff. 5 7 5 7 6.0 5 4 5 6 5.0 Assist one another. 5 5 4 5 4.8 5 3 4 6 4.5 Are collaborative. 5 5 3 5 4.5 4 4 4 4 4.0 Participation: Youth Are on-task. 3 7 5 7 5.5 4 4 5 5 4.5 Listen actively and attentively to peers and staff. 2 3 5 5 3.8 4 3 3 4 3.5 Contribute idea, opinions, and/or concerns to discussions. 3 3 3 5 3.5 4 4 5 3 4.0 Have opportunities to make meaningful choices. 5 1 5 5 4.0 3 3 5 5 4.0 Take leadership responsibibility/roles. 3 1 3 5 3.0 4 4 5 5 4.5 Relationship Building: Staff Use positive behavior management techniques. 5 5 5 5 5.0 6 2 5 5 4.5 Encourage the participation of all. 5 5 3 7 5.0 4 3 4 3 3.5 Show positive affect toward youth. 7 7 5 5 6.0 7 4 5 6 5.5 Attentively listen to and/or observe youth. 5 5 5 4 4.8 4 2 5 5 4.0 Encourage youth to share their ideas, opinions, and concerns about the content of the activity. 7 5 3 5 5.0 3 3 6 4 4.0 Engage personally with youth. 3 7 6 3 4.8 5 4 5 6 5.0 Guide positive peer interactions. 5 5 3 5 4.5 4 2 6 5 4.3 Instructional Strategies: Staff Communicate goals, purpose, expectations. 5 7 5 7 6.0 6 2 5 3 4.0 Verbally recognize youth’s efforts and accomplishments. 3 6 3 4 4.0 3 1 4 5 3.3 Assist youth without taking control. 5 6 5 5 5.3 5 2 5 5 4.3 Ask youth to expand upon their answers and ideas. 5 1 1 3 2.5 3 1 2 4 2.5 Challenge youth to move beyond their current level of competency. 5 3 1 3 3.0 3 1 5 3 3.0 Employ varied teaching strategies. 5 7 3 1 4.0 5 3 6 6 5.0 Plan for/ask youth to work together. 1 5 3 5 3.5 3 3 5 5 4.0 Program 1 Program 2 Executive Summary Research Questions Methodology Pre-work Training Observations Data Analysis Program One What the Staff Does Well Areas That Need More Attention Program Two What the Staff Does Well Ethical Analysis Norm of Caring Norm of Utilitarianism Suggestions for Improvement Learning Reflection Conclusion References