Original Research © The Author 2021. Published by the Coalition of Urban and Metropolitan Universities. www.cumuonline.org Metropolitan Universities | DOI 10.18060/23884 | February 15, 2021 35 Improving Workforce Readiness Skills among College Adult Learners through new Technologies: Lessons from Two Schools Sara B. Haviland1, Steven Robbins2, Vinetha Belur3, Gernissia Cherfrere4 , and David Klieger5 1,2,3,4,5 Academic to Career Research Center, Educational Testing Service Cite as: Haviland, S.B., Robbins, S., Belur, V., Cherfrere, G., and Klieger, D. (2021).Improving Workforce Readiness Skills among Adult Learners through new Technologies: Lessons from Two Schools. Metrpolitan Universities, 32(1), 35-53. DOI: 10.18060/23884 This is an open access article distributed under the terms of the Creative Commons Attribution License. Editor: Valerie L. Holton, Ph.D. Abstract Employers report struggles to find work-ready candidates who possess a desired combination of job-specific technical, general math and reading, and social and behavioral skills. Community colleges are ideally situated to address these shortages, delivering a trained local labor force and often collaborating directly with employers and regional development boards to ensure the employability of their students and the economic vitality of their regions. One tactic to address these shortages is to introduce trainings geared directly toward soft and basic skill development. This paper presents case studies of two schools that use technology-enhanced trainings to incorporate work-readiness skill trainings in career technical education programs that are (1) flexible and learner-centric due to technology-enhanced delivery, and (2) use evidence-based assessment and intervention strategies to promote soft skill and literacy expectations. The trainings were provided by a non-profit education research and measurement company, which served as developer and research partner for the community colleges. Results demonstrate adult learner satisfaction and commitment with these blended learning and mobile technology solutions. The authors discuss factors that facilitate training success, which include securing student buy-in and developing collaborative partnerships. Keywords: workforce development, adult learners, career and technical education, critical skill development, social and behavioral skills https://creativecommons.org/licenses/by/4.0/ © The Author 2021. Published by the Coalition of Urban and Metropolitan Universities. www.cumuonline.org Metropolitan Universities | DOI 10.18060/23884 | February 15, 2021 36 Introduction In the United States and globally, in cities and the regions they anchor, the knowledge and skills that employers, workforce preparation programs, and postsecondary institutions require have changed. Today, employers demand increased education and training as prerequisites for jobs (Casner-Lotto & Barrington, 2006; Hart Research Associates, 2015; Institute for a Competitive Workforce, 2012; Levy & Murnane, 2004). The increase in educational requirements is largely driven by skills gaps reported by employers. In addition to concerns about job-specific technical skills, employers report gaps in two critically important skill domains. First, for today’s complex technological economy, workers lack requisite cognitive and academic proficiencies (OECD, 2013). The second gap is a lack of proficiency across a broad array of behavioral competencies, which have been demonstrated to be instrumental to workplace success (Casner-Lotto & Barrington, 2006; Deming, 2017; National Research Council, 2011). These competencies sometimes have been referred to as “non-cognitive skills,” “socio-emotional skills,” “21st century skills,” “work readiness skills,” “soft skills,” or “employability skills” (see Campion et al., 2011; Parry, 1996). While technical skills help workers in specific jobs, noncognitive skills tend to be important for a wide variety of jobs. Taken together, these can be considered essential skills. Community colleges are well-positioned to be part of the solution in developing these competencies through affordable education that is often geared toward job-related skill development. With expanding portfolios in workforce development, the role of community colleges in directly meeting the skill demands of today’s employers is also expanding. This paper presents two community colleges serving urban students that recently worked with a partner institution (Educational Testing Service, or ETS, a non-profit educational organization) to pilot novel, technology-supported and evidence-based approaches to instilling critical competencies in adult learners. In both schools, the guiding research goal was to understand challenges and facilitating factors that accompanied the use of expanded remote technology to support essential skill development, and how these technologies were received by students. This paper highlights some of the implementation factors that supported success, including securing student buy-in and building organizational partnerships. Lessons across these programs can be used to inform future efforts to help students improve in these essential skill areas for workforce success. An Evolving Understanding of Essential Skills Cognitive competency and behavioral competency have different standing in formal education systems, community colleges included. Cognitive competencies are the basic knowledge and skills, such as reading, writing, math, and content/technical knowledge of facts and processes that are part of traditional academic curricula, typically the domain of developmental education. Behavioral competencies incorporate aspects of conscientiousness/ motivation that are distinguishable from cognitive skills, and are comprised of skills which primarily reflect © The Author 2021. Published by the Coalition of Urban and Metropolitan Universities. www.cumuonline.org Metropolitan Universities | DOI 10.18060/23884 | February 15, 2021 37 intrapersonal and interpersonal qualities such as initiative, resilience, teamwork, and responsibility. In some sense, different disciplines and industries may incorporate elements of cognitive and/or behavioral competencies as part of their education or career training. Educators have long understood that cognitive competencies are skills that can be developed over time, and community colleges endeavor to provide opportunities for students to develop these essential skills for career success. Three cognitive skill domains that comprise essential skills for both college and the workplace include prose/literacy, document/information use, and numeracy. These domains have been a central focus of cognitive skills and literacy assessment for more than 100 years (Kell & Lubinski, 2013; Kirsch, 2001; Resnick & Resnick, 1977) and continue to be critical to success in 21st century environments (OECD, 2013). As noted by the OECD, “at the most fundamental level, literacy and numeracy skills constitute a foundation for developing higher-order cognitive skills… and are essential for gaining access to and understanding specific domains of knowledge” (OECD, 2013, p. 56). Many students arrive at college prepared to function and expand in these skill domains; developmental education courses, a varied network of programs designed to prepare students academically for college- level work, are offered for those who require remediation. Unlike cognitive competencies, behavioral competencies have only more recently been understood as malleable skills that can be developed, and are perhaps equally important for personal and professional success in the growing global marketplace (National Research Council, 2011). Behavioral skills rarely are taught systematically on any educational level (Kirsch, Braun, Yamamoto, & Sum, 2007; National Research Council, 2012; cf. U.S. Department of Education, n.d.). Furthermore, the existing means, programs, and venues for assessing, developing, and supporting these competencies are not sufficient (National Research Council, 2012). Behavioral competencies programs are created and administered on an ad hoc basis and participation is often voluntary; the wide variety of methods deployed include workshops, courses, credentials, and career services programming (Noe, 2020). The existing evidence of valid and effective curricula and instruments to support career readiness skills development is limited; improving the evidence base of optimal implementation, program applicability and transportability between contexts, and the efficacy of cognitive and behavioral competency skill programs is an important task for workforce development today. Promising Applications of Technology to Support Community College Students Understanding that essential skills are malleable and valued by employers, it is important to identify systematic ways to improve proficiency among learners. To that end, the authors present some applications of technology that may be ideally suited to community college learners, by introducing flexibility into the learning process. As with any technology in the learning environment, it is important to examine our assumptions (Kruger-Ross & Holcomb, 2012). © The Author 2021. Published by the Coalition of Urban and Metropolitan Universities. www.cumuonline.org Metropolitan Universities | DOI 10.18060/23884 | February 15, 2021 38 Community colleges have diverse student bodies with specific learning challenges. Being predominantly adult learners and non-traditional students, community college students are often balancing competing demands of work and family that can hinder participation in traditional, classroom-based coursework. The two cases presented in this paper are deploying technology that extends learning beyond the traditional classroom, introducing flexibility to the learning process. Blended Learning Maximizes Remote Learning Online learning confers several benefits, such as (1) allowing for a scalable and standardized delivery of the assessment and curriculum, (2) immediate feedback and explanations for learners on the end-of-module knowledge checks, (3) the ability for instructors to monitor student progress in the online system, and (4) virtual access to instruction and practice opportunities outside of the classroom, an important capability particularly with adult learners. However, courses can also be designed from the bottom up to blend elements of both remote and classroom-based learning. This blended learning approach has been shown to be more effective than pure online learning or instructor-led courses, particularly at the postsecondary level (Means et al., 2013). Compared to pure online models, blended learning models leverage the role of the instructor, in-class group work, and whole-class discussion to facilitate engagement, critical thinking, and exposure to different viewpoints and perspectives, while allowing for opportunities for students to provide and receive in-person feedback (Garrison & Vaughan, 2008). This also facilitates what others have called a “situated” learning environment, an environment that is believed to be particularly effective for adult learners which promotes learning by allowing learners to bring their prior knowledge and experiences into the learning setting and construct new knowledge through interactions with their peers, their instructor and instructional materials and activities (Brown et. al, 1989). Finally, the in-class component of the curriculum allows instructors to provide additional support, including one-on-one support, to students who may be struggling with making progress in self-directed online content. Remote, Mobile-enhanced and Online Technology Allows Learners to Engage at Their Convenience One way to create a blended learning experience in an existing course is to add remote elements. App-based mobile technology offers two important pedagogical features: (1) instructional content presented in small “chunks” or “micro lessons” that allow for rapid absorption and rehearsal, and (2) practice test items with immediate feedback. With respect to the first feature, Fulantelli, Taibi, and Arrigo (2015) highlight the importance of content relevance, learner control of the learning process, and explicit and manageable activities in a proposed task- interaction learning analytics framework for mobile learning. These notions reflect the broader research on practice and successive relearning and rehearsal (c.f. Dunlosky & Rawson, 2015). © The Author 2021. Published by the Coalition of Urban and Metropolitan Universities. www.cumuonline.org Metropolitan Universities | DOI 10.18060/23884 | February 15, 2021 39 The mobile learning lessons and exercises used in these case studies reflect the critical pedagogical features highlighted by Fulantelli et al. (2015) in that they are based on both relevant and contextualized essential micro-lessons with training content that is based on evidence-based learning progressions that inform instructional content. Further, learners dictate when and where they participate in the training. With respect to the second feature, whether called retrieval practice, practice testing, or test-enhanced learning, it has been well-documented that practice with immediate feedback helps optimize learning (Dunlosky, Rawson, Marsh, Nathan, & Willingham, 2013; Roediger & Butler, 2011). While literature exists in various form regarding the use of technology in learning, or how to build essential skills, to date there is little work on how to use technology to build essential skills. In 2019, ETS partnered with two community colleges to pilot technology-enhanced essential skills trainings, working with each school to deliver foundational skills alongside their programs as part of two separate, but related research studies on essential skills development; one school delivering cognitive essential skill training, and the other delivering behavioral essential skill training. The ETS team evaluated these programs to understand challenges and facilitating factors that accompanied the use of expanded remote technology to support essential skill development, and how these technologies were received by students. Methods The two school pilots are approached as case studies, which were designed to determine efficacy and identify best practices for imparting essential skills to community college learners. In both cases, the training was provided at no expense to the schools by the research unit of ETS, which also paid implementation costs for the study. Both cases are designed as implementation evaluations and include data collections from students as well as faculty and staff of the institutions. Our data are drawn from several sources, including pre- and post-course surveys, interviews with faculty and administrators, and student focus groups Results are shared by school, demonstrating the challenges and possibilities of efforts to bring behavioral and cognitive skill development to these students with blended learning models. The two programs are profiled in Table 1. Case Study 1 occurred in the continuing studies program at a large northeastern community college serving a state capital with roughly half minority enrollment, largely black. Case Study 2 occurred in two different departments and courses, the Department of Adult Education and the School of Arts & Sciences’ College Readiness Department, at a very large two-year Midwestern college that serves a large proportion of career technical education students in a major city, with a majority enrollment of minorities, largely Hispanic. Results are framed around the key stakeholders in each system: the students, and the instructors and college personnel. The authors provide discussion about each case before turning to general discussion across cases. © The Author 2021. Published by the Coalition of Urban and Metropolitan Universities. www.cumuonline.org Metropolitan Universities | DOI 10.18060/23884 | February 15, 2021 40 Table 1. Profile of two case studies Case 1 Case 2 Region Northeast Midwest School size 5,000-9,999 Over 10,000 Essential skills covered Behavioral Cognitive Technology Web-based lessons Mobile app Program location Standalone, non-credit course Embedded into developmental education and adult learning courses Case Study 1 This case study featured a standalone eight-week course offered through the school’s continuing studies program during the Summer and Fall terms of 2019. To participate in this free course, the only pre-requisites were (a) to be at least 18 years old and (b) to have either a high school diploma or high school equivalency credential. Topics included initiative, perseverance, responsibility, flexibility, resilience, teamwork, citizenship, customer service orientation, problem solving, and ingenuity. Participants self-enrolled in either an enhanced self-directed or a more fully blended learning version of the course; both featured the same content and were run concurrently by the same instructor. In both versions, students were expected to attend the first and last session in-person. During the first session, students completed a personality assessment and reviewed the score report with the instructor, received an overview of the course design and expectations, and participated in data collection via student experience surveys. During the last session, students de-briefed with the instructor on their experiences and participated in post- course data collection via surveys and/or focus groups. The enhanced self–directed cohort completed eight online training modules, along with any pre- work or homework assignments, independently and at their own pace, and therefore had a “blend” that favored more flexible and independent learning. This was the less popular version, with only three students enrolling in the first semester the program was offered, and one in the second. The fully blended learning cohort completed the online training modules as homework on a weekly basis and participated in a one-hour in-person session at the institution, and therefore had a “blend” that comprised roughly half flexible and independent learning, and half in-person engagement and support. The instructor reviewed portions of the online modules and geared most of the class time towards engaging in activities, facilitating discussions and extending the lesson with relevant handouts. Both versions of the course featured brief knowledge checks upon the completion of each online learning module, which were multiple choice and demonstrated the cognitive skill that was the subject of each lesson. The course experienced low enrollments that resulted in only 13 student completions across two class cohorts. The instructor and administrator each shared their perspectives in a formal exit interview and through ongoing check-ins with the research team. Across cohorts, the © The Author 2021. Published by the Coalition of Urban and Metropolitan Universities. www.cumuonline.org Metropolitan Universities | DOI 10.18060/23884 | February 15, 2021 41 race/ethnicity of the 13 participants was White (31%, n=4), Hispanic (31%, n=4), Asian (23%, n=3), and Black/African American (15%, n=2). The participants were primarily female (77%, n=10) and held at least an associate’s degree (69%, n=9). Initially during the summer 2019 course, 10 students registered for the WorkFORCE Program for Career Development course, which included seven students for the blended learning course and three students for the enhanced self-directed course. However, only five blended learning students and three enhanced self-directed students (a) consented into the study (b) attended the first session, and (c) proceeded with attending the subsequent courses. For the fall 2019 course, nine students attended the first day of class, with five students persisting to the end of the course. All students who completed the course participated in the post-course survey, but with only 13 respondents our focus was on the qualitative-based open-text response items rather than numerical analyses. Case Study 2 This case study integrated a mobile learning training platform (the app) for cognitive skill development as a required component of existing developmental education and adult education programs. This consisted of 150 micro-lessons plus practice items to be delivered on learners’ mobile phones via a text messaging app. These lessons are tied directly to the essential literacy domains in the cognitive assessment, which is an assessment of abilities in prose, documentation, and quantitative literacy. Students were given a cognitive assessment of their literacy skills at the beginning of the fall 2019 academic semester, the results of which were shared with the instructors. Students then completed at least eight hours of mandatory homework on a mobile training app over the course of 8 weeks. At the conclusion of the eight weeks, the cognitive assessment was given again, along with a brief survey of student perceptions of the program. The two classes were a non- credit (pass or repeat) Language Arts GED/Adult Education class held twice a week for 3.5 hours at a time and an introductory college reading and writing Developmental Educational class, also non-credit, held twice a week for 2.25 hours at a time. Students completed at least eight hours of mandatory homework on a mobile training app over the course of 8 weeks. At the conclusion of the eight weeks, the cognitive assessment was given again, along with a brief 31- item survey of student perceptions that was administered online. The study team also conducted two 30-minute focus groups to inquire about students’ hands on experiences in using the app and their perspectives on the usefulness of the app in improving their literacy skills; and a 30-minute phone interview with each instructor. The focus groups, led by the deans of the two departments, were audio-recorded and analyzed by the research staff. The two instructors played a key role in the implementation of the training intervention by acting as moderators or brokers of the process. First, they provided feedback to the research team on all aspects of the training materials and process, including the assessment process, the score report, and alignment of the mobile app content with their teaching goals. They did this both during the © The Author 2021. Published by the Coalition of Urban and Metropolitan Universities. www.cumuonline.org Metropolitan Universities | DOI 10.18060/23884 | February 15, 2021 42 implementation and also in interviews after its completion. They also gauged students’ reactions to the training intervention and keeping the research team abreast of the how the intervention was being received. Instructors sent emails, at times on a daily basis, to share their feedback on students’ reactions and the pulse of the classroom. Administrators also provided feedback to the research team through check-in emails and reviews of the initial project report. In Case Study 2, 34 students enrolled across both classes. with 21 CTE learners (a) consenting to the study, (b) attending the first session, and (c) proceeding with attending all subsequent courses. More than half of the participants (57%) identified as Hispanic. The rest were Black/African American (24%) or White (14%). Results: Instilling Essential Skills in Community Colleges The discussion is framed around the key stakeholders in each system: the students, and the instructors and college personnel. We provide discussion about findings specific to each case before turning to the general discussion across cases. Case Study 1: Voluntary Behavioral Competencies Training In the following results, the authors explore the experiences and perspectives of the various stakeholders (i.e., students, administrators, and instructor) to assess the quality of the training, and then explore factors that created challenges for implementation of the course. Unless otherwise noted, both the strengths and potential areas for improvement are echoed across the various data sources and course offerings (i.e., summer and fall 2019, enhanced remote versus a more blended learning model). Student Perspectives The following findings are drawn from open response items on post-course student surveys. Motivations for Participation. Participants reported that they were motivated to register for the training because they viewed it as a preparatory course for career development to start or re-enter the workforce or an opportunity to enhance their job skills. Given that the training program provides scaffolded material in the curriculum that supports participants in these areas, it appears that the participants had goals and expectations that were aligned to the overall objective of the course. © The Author 2021. Published by the Coalition of Urban and Metropolitan Universities. www.cumuonline.org Metropolitan Universities | DOI 10.18060/23884 | February 15, 2021 43 For instance, a few students shared in student surveys: o “I want to learn to prepare for working. I want to learn how to solving problem [sic] and communicate with people in work.” o “I'm looking for a job now and would like to be prepared better in advance.” o “I think this is my first step to start my career.” According to survey data, if given the opportunity to retake the course, the majority of participants would be primarily motivated by the incentive of a certificate of completion or feedback on their skills. Satisfaction with Content and Delivery. Overall, student respondents were satisfied with the program’s socio-emotional benefits and work readiness preparations, reporting increased confidence in their ability to use soft skills such as teamwork and responsibility. They also found the online modules easy to manage, informative, and visually stimulating. Students felt more equipped with work readiness skills to pursue new forms of employment and more confident engaging in the hiring process. For example, some students shared: “The whole training with all the modules are really helpful for people to get prepared for [an] interview. And I think this will be very beneficial for the students when they graduate college.” “So, I’ve never worked before. I mean, it’s been like ages [since] I stopped working. I was a little bit nervous how will I get trained, and how will I start my first job and how will I go for interview, what they’ll ask me and all that stuff? But after coming through this course, it made me really more confident. I was happy throughout the course. I mean, I’ve learned many things and I’m ready for a new job.” The program offered these students targeted training that increased their confidence as they prepared for the job market. In another instance, a blended learning student shared that the module transcriptions and captions were very beneficial in helping the student comprehend and learn the content, which was an advantage of the format because English was not the student’s primary language. Instructional Quality and Assessment. When reflecting on the instructional delivery of the course content in summer 2019, both the blended learning and self-directed students also shared an appreciation for the high level of interaction and engagement the instructor provided. The blended learning environment featured opportunities for students to share experiences and learn from one another. For the self-directed group the instructor also provided weekly email check-ins with the students and gave the students feedback. The instructional assistance was so well- received that the self-directed students recommended the instructional support be increased to twice per week. In reference to the Knowledge Check assessments, all the respondents either © The Author 2021. Published by the Coalition of Urban and Metropolitan Universities. www.cumuonline.org Metropolitan Universities | DOI 10.18060/23884 | February 15, 2021 44 agreed or strongly agreed that they were comfortable with the process of completing the assessment and the assessment results made sense to them. In this same vein, all the participants agreed that the assessment gave them a better sense of what they can work on to be a stronger student or employee. Room for Improvement. While students were generally satisfied with the course offering, they also provided suggestions for areas needing improvement. For instance, multiple students suggested lengthening the blended learning course from one hour to two hours, because they believed the dialogue and constructive feedback with colleagues and the instructor were very useful, but often limited due to time constraints. Students also recommended a follow- up course be offered to students so that students could continue to access and learn from the modules. Additionally, survey data showed that participants also identified technical problems with the internet modules and issues of alignment with career values as areas for improvement. Offering possible recommendations to strengthen the course, participants shared a desire to engage with more realistic examples, engage directly with career services, and receive hardcopies of the course materials, indicating they craved a learning experience that was further contextualized, a promising model (Fulantelli et al., 2015). Instructor Reflections Support More Blended Model. Overall, the instructor shared that his primary goal was to build students’ confidence with each soft skill introduced in the program. For instance, the instructor shared, “If they [students] can come away feeling more confident about themselves as an employee and how they handle situations at work, then I think, they got out of it what they should have.” During the eight-week course, the instructor found the implementation of the curriculum, correspondence, and resources (e.g., modules, videos, and pre-work articles) well-prepared and useful. However, he encountered inconsistent student attendance, and noted that low accountability due to the voluntary nature of the program was an ongoing challenge. When students attended class, time constraints also posed as a challenge because the 60 minutes allotted did not provide enough opportunity to engage in discussions. The instructor also noted that the self-directed students had low student participation in written discussions, which he found somewhat problematic. The instructor also reflected on the possibility of converting the course to be credit bearing, recommending that the type of course and number of students within the selected course be considered to determine the adequate length of time for the training. If engagement and student participation is important, the instructor felt that increasing the program length to more than 60 minutes would give students the opportunity to engage in meaningful conversations and share their experiences. The instructor also felt that there should also be higher expectations on the student workload, more clearly defined logistics on how it will be implemented within a class © The Author 2021. Published by the Coalition of Urban and Metropolitan Universities. www.cumuonline.org Metropolitan Universities | DOI 10.18060/23884 | February 15, 2021 45 setting, and an expansion on the importance of the program. Given the instructor’s feedback, the blended learning version of the course appeared to be the optimal condition. Supporting Equity. In interviews, the program administrator shared that she was initially unsure about the relevance of the training, which was novel for the continuing studies department. However, after seeing the course in action and receiving positive feedback from the instructor and students, she felt that the training was very beneficial to the department. Since the non-credit bearing centers are revenue-driven, the administrator noted that it could have been difficult to implement the program if there were not the funds provided to assist with implementing the course. Students represented various age groups and life experiences, and therefore work and educational experiences also varied. The administrator remarked that the program provided an important avenue for students to become more marketable, sharing, “Some are new immigrants; some are individuals who are restarting their careers maybe after a long gap. So, they come to non-credit to get these types of trainings to get back into a marketable individual in the marketplace. Sometimes, we have individuals who are struggling to find a way as to why they are not progressing in professional world despite of having the training. So, this could be the missing piece for them.” Given the students the administrator noted to benefit from the program, it was a useful way to close key equity gaps in the job market. Beyond the credit bearing courses, the administrator also recommends offering the program to bilingual programs to help bridge the cultural gap. Case Study 2: Integrated Cognitive Skills Program In the following results, the authors explore the experiences and perspectives of the various stakeholders (i.e., students, administrators, and instructor) to assess the quality of the training, and then explore factors that created challenges for implementation of the course. In general, students became more confident in their reading skills because of using the app, and many also noted the ease of accessing the app’s content at any time. Instructors felt the students were excited to use the app for homework and felt the content on the app aligned well with their teaching goals. Satisfaction with Content and Delivery. Participants cited three main ways they benefited from using the app during the semester including (a) the ease of learning anywhere/anytime, (b) the feedback and explanations provided by the app when an answer was incorrect, which helped to increase their reading skills and reading comprehension, and (c) the skills the app taught them in scanning the paragraph for answers to their questions and looking for keywords. With respect to ease of learning, eleven of the adult learners reported they used the app for short chunks of time, for instance, after their workday, while cooking, while waiting at the doctor’s office, or when © The Author 2021. Published by the Coalition of Urban and Metropolitan Universities. www.cumuonline.org Metropolitan Universities | DOI 10.18060/23884 | February 15, 2021 46 relaxing after dinner. On the other hand, six adult learners used the app for extended periods of time, and four learners said they used it both ways. The learners reported they enjoyed the convenience the app gave them to learn the way they preferred and when they preferred. With respect to improving reading skills and scanning skills, nine of the adult learners reported feeling like the app was very useful in increasing these skills. In terms of text messages that were sent to participants after a certain period of non-activity, eleven students agreed that the text messages motivated them to open the app and work on the practice lessons. App and Assessment Experiences. With respect to the pretest, a majority of the 21 student survey respondents indicated that seeing the score report from their initial pre-course assessment increased their desire to improve themselves. Eight students were satisfied with their scores, while eleven were neutral. With respect to the training app, participants generally reported they liked using the app for homework. Most participants (n=18) felt that doing the practice lessons on the app helped them to improve their reading or reading comprehension skills, and seventeen participants reported they liked using the mobile app because it allowed them to practice and learn anywhere/anytime using their cell phone. A majority of students indicated they would welcome additional homework assignments on an app similar to the one used this semester. Instructors Moderate the Learning Experience. Instructors acted as advocates for their students during the intervention process. For instance, when distributing pretest score reports to students, both instructors thought of ways to mitigate potential negative self-perceptions due to low scores on the assessment. One instructor indicated she buffered students by saying “Don’t take this personally, but have a goal in mind that…this is where you’d like to be at the end of semester.” Similarly, the instructors cautioned the research team that setting an expectation that the assessment would take 2 or 2.5 hours rather than 1.5 hours would allow students to prepare themselves better mentally and also prevent discouragement. One instructor noted, “Some might have felt discouraged about their abilities since we told them 1.5 hours and it was taking them longer.” The instructors also performed in a motivational function, for instance, motivating students to perform their best on the pretest by telling them their performance on the pretest would determine the kinds/types of activities they were given the rest of the semester on the mobile app. Equity Considerations and Room for Improvement. Program administrators reported major differences in students’ eagerness and interest to participate in the focus groups by class, with the GED/Adult Education students more motivated and eager to share their experiences, and the Developmental Education class less motivated and more reluctant to talk. This same theme of more motivation from the GED/Adult Education students to work on the app and less motivation from the Developmental Education students was also observed by the instructors and research team throughout the implementation of the program. The administrators elaborated on several differences between the two groups of students including age difference with half of the students from the GED/Adult Education being over age 30 which might account for their comfort in © The Author 2021. Published by the Coalition of Urban and Metropolitan Universities. www.cumuonline.org Metropolitan Universities | DOI 10.18060/23884 | February 15, 2021 47 speaking up in the focus groups. The administrators also shared that the GED/Adult Education program is grant-funded and as such there is flexibility to give these students a tremendous amount of support and structure including more advising hours, more communication with students, a structured attendance policy, and more professional development experiences. However, the administrators also noted that the two groups were similar in other ways, including “race, SES, dispositional barriers (self-efficacy, motivation, consistency, perseverance, self- awareness, attention span), life challenges, learning challenges." Implementation Lessons Across Programs Overall, these Case Studies demonstrated a positive experience for students who persisted in these technology enhanced, blended-learning experiences. Case Study 1 behavioral skills training was well received by the instructor, students, and administrator as a non-credit bearing course at the community college. The two areas that resonated across the three stakeholders was the usefulness of the program towards (a) increasing work readiness skills and (b) strengthening the confidence of students in employing those behavioral competencies. With the possibility of scaling up the implementation of soft skills training across multiple classes and institutions, there are some implications that merit consideration. In this case study, the administrator specifically selected the instructor, because his interests and skill sets aligned with the overall objective the program. However, in scaling up soft skill training, programs may be introduced to instructors as required top-down initiatives. Therefore, developing avenues to collaborate with or incentivize instructors may build stronger instructor buy-in. Case Study 2 aimed to pilot an essential cognitive skills assessment plus app solution intended to improve essential literacy skills via immediate feedback, in two community college classrooms with career technical and adult learners. Some of the most basic and important findings from this study are that across both classes, a majority of learners looked forward to seeing the results of their initial assessment, found the score report encouraging, and in at least one class, were looking forward to using the app, asking about it multiple times. Learners generally enjoyed using their mobile devices and apps as part of their course assignments. Limitations to this study include the absence of post-course data for students who dropped the course, a common challenge in educational training program evaluation that may have been compounded by the absence of incentives (both the voluntary nature of the program, and a lack of participant incentives for the study). Future studies are planned to shift these programs into existing required curriculum in both credit and non-credit programs and add participant incentives to increase research participation, even among non-completers. Even with only sparse data, in comparing the cases, notable lessons about implementation included the role of student buy-in in implementation success, the role of third party support in the implementation process. The authors discuss each in turn. © The Author 2021. Published by the Coalition of Urban and Metropolitan Universities. www.cumuonline.org Metropolitan Universities | DOI 10.18060/23884 | February 15, 2021 48 Student Buy-in and Implementation Success In both Case Study 1 and 2, the general student enthusiasm for content and delivery were encouraging, as learner motivations can predict initial skill acquisition (Bauer et. al, 2016). While both programs were well-received by the participants who completed them, recruitment and retention were major challenges for Case Study 1, the voluntary program, while Case Study 2 was much more successful. Key differences between the programs that likely contributed to this challenge included the voluntary nature of the program and the tuition free model, which meant that students did not have any financial stake in the program. In Case Study 1, voluntary participation may be related to student attrition and low student participation, due to the low accountability of students to attend or complete the sessions. This was a source of frustration for the instructor and implementation team. In order to maintain high fidelity of implementation and sustain student participation in a scalable model, attaching the program to a pre-existing course as required homework may be a more viable option. Incorporating an essential skills curriculum into an extant credit-bearing course, as was done with Case Study 2, can ensure that all learners experience the curriculum and will be motivated to complete the training activities. Integration provides opportunities for learning and practicing the behavioral competencies in the context of technical content related to the learner’s career aspirations. As demonstrated by Knowles (1984), placing essential skills instruction into context is best practice for designing adult learning environments that stress the criticality of the immediate pertinence of the topics for the learner and their career. Further, incorporation of essential skills curriculum into courses promotes a situated learning environment particularly effective for adult learners (Brown, Collins, & Duguid, 1989). Learners bring their prior knowledge and experiences into the learning setting and construct new knowledge by working with their peers, their instructor, and the instructional materials and activities. By integrating essential skills training as a homework task, as in Case Study 2, all stakeholders appear to have a more seamless experience. Instructors are not burdened with the loss of instructional time or with the need to reframe the scope of their predesigned course. Students may feel more incentivized to complete the entire program as opposed to only partially completing the tasks and Knowledge Check assessments. Students do not have to allocate additional time to attend a separate course, but instead the additional training is infused within their scheduled coursework. In working with field partners, the research team observed that, independent of the context and content of an actual course, learners were not likely to appreciate the importance of behavioral competencies. Thus, they might be disinclined to enroll in a standalone development course on behavioral competencies, particularly if completing the course does not earn the student additional credits towards their certificate. Integration provides opportunities for learning and © The Author 2021. Published by the Coalition of Urban and Metropolitan Universities. www.cumuonline.org Metropolitan Universities | DOI 10.18060/23884 | February 15, 2021 49 practicing the behavioral competencies in the context of technical content related to the learners’ career aspirations. This allows the instructors and learners to relate the online content to content and issues, both career-related and technical, covered in coursework, allowing learners to reflect on how the skills apply both in the classroom as well as within their chosen careers. Furthermore, facilitating implementation factors may include whether the class is for credit and for a grade, or is non-credit bearing with only pass/fail options. Unless completing the homework and actively participating is tied to whether a student passes or fails, students may be able to skip homework and class sessions and still pass the class. On the other hand, students with below- average but not very low reading abilities, in a class where the homework/program is required for a grade, and with an instructor whose attitude reflects that they believe the program is do-able by their students, might have better outcomes with respect to student motivation and interest in the program. Third-party Engagement and Scalability In considering long-term scalability and sustainability for these programs, it is important to acknowledge the role that third party engagement played. This factor was cited by instructors and administrators in both programs as a facilitator for program success. Factors that made these partnerships work included the transparency of ETS’s expectations and goals, frequent communication, joint planning, and a commitment to feedback-formative evaluation- improvement. Members of the ETS team worked closely with administrators in the planning phase to determine implementation design and recruitment. ETS project managers offered support and troubleshooting as needed for program technology and implementation, and worked closely with instructors throughout the launch and implementation of the trainings. Community colleges enter into partnerships with local organizations to expand their impact, connect to employers, and/or bring additional resources into their programs (e.g. Haynes et al., 2018, . Typically, this is done through contract trainings, technology centers, workforce development partnerships, welfare-to-work, or business-based scholarships (Kisker and Carducci, 2003). In this study partnership the pairing is somewhat different, as the college’s partner is the research arm of a non-profit and non-governmental entity. However, this introduced resources to the school that removed costs typically associated with introducing novel curricula/pedagogy, which was a factor in the success of the programs. The resource intensity of this model threatens scalability; future studies are planned to determine if the programs would succeed without the same level of third party resources and expertise, and what adjustments can be made to ensure long-term program successes. © The Author 2021. Published by the Coalition of Urban and Metropolitan Universities. www.cumuonline.org Metropolitan Universities | DOI 10.18060/23884 | February 15, 2021 50 Conclusion Though small in scale, the two case studies demonstrated student enthusiasm for novel, technologically-supported forms of essential skills training, as students in both programs reported general satisfaction with the trainings and increased confidence in their abilities. However, the form of implementation mattered; in particular, integration of the curriculum into an existing, mandatory course, as opposed to a stand-alone, voluntary, or non-credit bearing course, increased motivation for all students to engage in the curriculum and increased efficacy through providing opportunities to learn and practice in context. A strong working relationship between partners appeared to facilitate program success, and frequent communications and joint planning go a long way to ensure that relationship. Given the small sample of students and high incidence of dropout in Case Study 1, it is imperative that research continue to fortify our understanding of these novel programs; further action research is needed to understand the impact of mobile learning modality relative to classroom and homework-based instruction and how to embed co-curricular activities that promote essential skill work readiness benchmarks in existing CTE curriculum. The results of these case studies suggest best practices to inform these efforts and provide necessary structure to support student success in essential skills development. These promising results warrant further study on a larger scale. © The Author 2021. Published by the Coalition of Urban and Metropolitan Universities. www.cumuonline.org Metropolitan Universities | DOI 10.18060/23884 | February 15, 2021 51 References Bauer, K.N., Orvis, K.A., Ely, K. & Surface, E.A.(2016). Re-examination of Motivation in Learning Contexts: Meta-analytically Investigating the Role Type of Motivation Plays in the Prediction of Key Training Outcomes. Journal of Business and Psychology 31, 33– 50. https://doi.org/10.1007/s10869-015-9401-1 Brown, J. S., Collins, A., & Duguid, P. (1988). Situated cognition and the culture of learning. Educational Researcher, 18(1), 32-42. https://doi.org/10.21236/ada204690 Campion, M. A., Fink, A. A., Ruggeberg, B. J., Carr, L., Phillips, G. M., & Odman, R. B. (2011). Doing competencies well: Best practices in competency modeling. Personnel Psychology, 64, 225–262. https://doi.org/10.1111/j.1744-6570.2010.01207.x Casner-Lotto, J., & Barrington, L. (2006). Are they really ready to work? Employers’ perspectives on the basic knowledge and applied skills of new entrants to the 21st century U.S. workforce. The Conference Board, Partnership for 21st Century Skills, Corporate Voices for Working Families, and the Society for Human Resources Management. https://files.eric.ed.gov/fulltext/ED519465.pdf Deming, D.J. (2017). The Growing Importance of Social Skills in the Labor Market. The Quarterly Journal of Economics, 132(4), 1593–1640. https://doi.org/10.1093/qje/qjx022 Dunlosky, J., & Rawson, K. A. (2015). Practice tests, spaced practice, and successive relearning: Tips for classroom use and guiding students’ learning. Scholarship of Teaching and Learning in Psychology, 1(1), 72-78. https://doi.org/10.1037/stl0000024 Dunlosky, J., Rawson, K. A., Marsh, E. J., Nathan, M. J., & Willingham, D. T. (2013). Improving students’ learning with effective learning techniques: Promising directions from cognitive and educational psychology. Psychological Science in the Public Interest, 14(1), 4-58. https://doi.org/10.1177/1529100612453266 Fulantelli, G., Taibi, D., & Arrigo, M. (2015). A framework to support educational decision making in mobile learning. Computers in Human Behavior, 47, 50-59. https://doi.org/10.1016/j.chb.2014.05.045 Garrison, D. R., & Vaughan, N. D. (2008). Blended learning in higher education: Framework, principles, and guidelines. Jossey-Bass. https://doi.org/10.1002/9781118269558 https://doi.org/10.1007/s10869-015-9401-1 https://doi.org/10.21236/ada204690 https://doi.org/10.1111/j.1744-6570.2010.01207.x https://files.eric.ed.gov/fulltext/ED519465.pdf https://doi.org/10.1093/qje/qjx022 https://psycnet.apa.org/doi/10.1037/stl0000024 https://doi.org/10.1177/1529100612453266 https://doi.org/10.1016/j.chb.2014.05.045 https://doi.org/10.1002/9781118269558 © The Author 2021. Published by the Coalition of Urban and Metropolitan Universities. www.cumuonline.org Metropolitan Universities | DOI 10.18060/23884 | February 15, 2021 52 Golich, V.L., Hanyes, S., & Kreidler, S. (2018). Using Public Private Partnerships to Accelerate Student Success. Metropolitan Universities. 29 (3). https://doi.org/10.18060/22324 Hart Research Associates. 2015. Falling Short? College Learning and Career Success. Washington, DC: Association of American Colleges and Universities. https://files.eric.ed.gov/fulltext/ED519465.pdf Institute for a Competitive Workforce. (2012, September). Help wanted 2012: Addressing the skills gap. U.S. Chamber of Commerce. https://www.uschamberfoundation.org/sites/default/files/publication/edu/HelpWanted%2 02012.pdf Kell, H. J., & Lubinski, D. (2013). Spatial ability: A neglected talent in educational and occupational settings. Roeper Review, 35(4), 219-230. https://doi.org/10.1080/02783193.2013.829896 Kirsch, I. S. (2001). The international adult literacy survey (IALS): Understanding what was measured (ETS Research Report RR-01-25). Educational Testing Service. https://doi.org/10.1002/j.2333-8504.2001.tb01867.x Kirsch, I., Braun, H., Yamamoto, K., & Sum, A. (2007, January). America’s perfect storm: Three forces changing our nation’s future. Educational Testing Service. https://www.ets.org/Media/Education_Topics/pdf/AmericasPerfectStorm.pdf Kisker, C. B., & Carducci, R. (2003). UCLA community college review: community college partnerships with the private sector--organizational contexts and models for successful collaboration. Community College Review, 3, 55-74. https://doi.org/10.1177/009155210303100304 Knowles, M. (1984). Andragogy in action: Applying modern principles of adult learning. Jossey- Bass. https://doi.org/10.22230/cjc.1986v12n1a376 Kruger-Ross, M.J. & Holcomb, L.B. (2012). Educational Technology as a Subversive Activity: Questioning Assumptions Related to Teaching and Leading with Technology. Metropolitan Universities, 23(2), 129-141. Levy, F., & Murnane, R. J. (2004). The new division of labor: How computers are creating the next job market. Princeton University Press. https://doi.org/10.1515/9781400845927 https://doi.org/10.18060/22324 https://files.eric.ed.gov/fulltext/ED519465.pdf https://www.uschamberfoundation.org/sites/default/files/publication/edu/HelpWanted%202012.pdf https://www.uschamberfoundation.org/sites/default/files/publication/edu/HelpWanted%202012.pdf https://doi.org/10.1080/02783193.2013.829896 https://doi.org/10.1002/j.2333-8504.2001.tb01867.x https://www.ets.org/Media/Education_Topics/pdf/AmericasPerfectStorm.pdf https://doi.org/10.1177/009155210303100304 https://doi.org/10.22230/cjc.1986v12n1a376 https://doi.org/10.1515/9781400845927 © The Author 2021. Published by the Coalition of Urban and Metropolitan Universities. www.cumuonline.org Metropolitan Universities | DOI 10.18060/23884 | February 15, 2021 53 Means, B., Toyama, Y., Murphy, R., & Bakia, M. (2013). The effectiveness of online and blended learning: A meta-analysis of the empirical literature. Teachers College Record, 115, 1-47. National Research Council. (2011). Assessing 21st century skills: Summary of a workshop. The National Academies Press. https://doi.org/10.17226/13215 National Research Council (2012). Education for life and work: Developing transferable knowledge and skills in the 21st century. The National Academies Press. https://doi.org/10.17226/13398 Noe, R. (2020). Employee Training and Development (8th edition), McGraw-Hill: New York. Organization for Economic Co-operation and Development. (2013). OECD skills outlook 2013: First results from the survey of adult skills. OECD Publishing. http://dx.doi.org/10.1787/9789264204256-en Parry, S. B. (1996). The quest for competencies. Training, 33(7), 48–54. Resnick, D., & Resnick, L. (1977) The nature of literacy: An historical exploration. Harvard Educational Review, 47(3), 370-385. https://doi.org/10.17763/haer.47.3.27263381g038222w Roediger, H. L., & Butler, A. C. (2011). The critical role of retrieval practice in long-term retention. Trends in Cognitive Sciences, 15(1), 20-27. https://doi.org/10.1016/j.tics.2010.09.003 U.S. Department of Education. (n.d.). Every student succeeds act (ESSA). https://www.ed.gov/essa https://doi.org/10.17226/13215 https://doi.org/10.17226/13398 http://dx.doi.org/10.1787/9789264204256-en https://doi.org/10.17763/haer.47.3.27263381g038222w https://doi.org/10.1016/j.tics.2010.09.003 https://www.ed.gov/essa Sara B. Haviland1, Steven Robbins2, Vinetha Belur3, Gernissia Cherfrere4 , and David Klieger5 Results: Instilling Essential Skills in Community Colleges Case Study 1: Voluntary Behavioral Competencies Training Case Study 2: Integrated Cognitive Skills Program Implementation Lessons Across Programs Kirsch, I. S. (2001). The international adult literacy survey (IALS): Understanding what was measured (ETS Research Report RR-01-25). Educational Testing Service. https://doi.org/10.1002/j.2333-8504.2001.tb01867.x