Frontiers of Contemporary Education ISSN 2690-3520 (Print) ISSN 2690-3539 (Online) Vol. 2, No. 2, 2021 www.scholink.org/ojs/index.php/fce 42 Original Paper Linking Budgeting with Computational Thinking Pedagogy: Program Theory, Performance, and Budgeting Michael Lindsay Hall 1 1 Roger Williams University, Bristol and Providence, Rhode Island, United States Received: August 10, 2021 Accepted: August 19, 2021 Online Published: September 25, 2021 doi:10.22158/fce.v2n2p42 URL: http://dx.doi.org/10.22158/fce.v2n2p42 Abstract The pedagogy involved with preparing and delivering an analytically based course must contend with a number of important limitations or challenges. The challenges/limitations include needing a context for the use of the analytics being taught; others include where best to embed analytic courses in degree curriculum, determining content and delivery along with a number of additional limitations. A context can be created for these courses by establishing a base of usefulness of the course content and how it relates to other courses and to professional applications. However, one useful approach for a budgeting course is to put the analytics in a context of production and performance. These two significant elements of any problem-solving organization finance and budgeting process are significant features of teaching a course in budgeting. The article presented here is an illustration of a context-based approach along with features of pedagogy based in computational thinking which can be used to operationalize course elements while overcoming other salient limitations for analytic courses. The exemplar of a budgeting course is posed as an example. Keywords computational thinking, budgeting, program theory, program design 1. Introduction Analytically based courses are, by their very nature, challenging to develop and deliver. Courses requiring arithmetic and/or mathematical skills are the ones that offer particularly unique kinds of challenges. Those challenges begin with contextual limitations. Some of the limitations are presented by class participants. Further limitations would be content for linking with other courses to provide a scaffolding for learning and for deploying analytics professionally. A pedagogical approach to deal with this kind of problem can begin by establishing a context of usefulness of the arithmetic or math applications being taught. Courses such as production design, operations research, finance, accounting, www.scholink.org/ojs/index.php/fce Frontiers of Contemporary Education Vol. 2, No. 2, 2021 43 Published by SCHOLINK INC. statistics, program evaluation, or budgeting which rely heavily on math would be the kind of courses to have the kinds of contextual limitations which potentially reduce the likelihood of maximum learning and intention to transfer to professional use. Courses intended to build problem solving skills are found in MBA, MPA, and Health Administration professional degree programs, among others. An example of the approach to create a context for mitigating context limitations follows using an MPA budgeting course as a point of reference for a way to overcome the challenges in analysis-based courses. 2. Literature Review The budgeting course in Public Administration Degree programs is probably best understood as a pivotal course. It is pivotal because budgets exist for programs to function. Programs are what MPAs are being educated to manage. Courses in such Public Administration curricula are also intended for nonprofit applications as well as health care organizations. There are such courses in MBA degree programs for people interested in nonprofit management. So important is the program and budget linkage that, according to Howard Frank (2002, p. 147) a key component of the budget course he discusses is performance or simply put, how a government program works and needs to work. An apt analogy here might be one used by B. Guy Peters in his book American Public Policy, in which he refers to programs as engines of public policy. His book title, coincidentally, contains the subtitle “Promise and Performance” (Peters, 2019). Program performance is directly linked to budgets for those programs. Operations research for manufacturing or even services would offer a similar academic footing. Operations research is about production and performance which are at the heart of any service, program, or other organized, purposeful set of tasks intended to create an organizational result. However, traditional budgeting with its line items does not clearly offer a complete picture of production. Thus, a budgeting course must navigate the limited information base of line items to get to performance as a topic. It is also not uncommon for the production process to be unconnected to the budgeting process, with little input to decisions other than “budget requests”. The result is students take course a course for which they have only limited individual context and, if they are “in service” learners, they ordinarily do not encounter performance and budgets actually linked in a meaningful way. The course structure and discussion which follows is one way to take on this teaching task of creating linkages in thinking to programs/production from the important context of the precursors to budgeting. 2.1 Programs/Production Come First Since the engine metaphor is so workable for the understanding of what drives government policy or nonprofit or healthcare programs, it is only appropriate to recognize that engine design is critical to performance and fuel usage. An engine can only perform the way it was designed to operate and its fuel usage is also dependent on the design; equally the case with programs. Programs cannot create effects or results beyond their design. Consequently, program design is no small consideration when developing budgets. Attention to the “best that we know” for program design is essential for www.scholink.org/ojs/index.php/fce Frontiers of Contemporary Education Vol. 2, No. 2, 2021 44 Published by SCHOLINK INC. performance and the budgets that fuel it. The same would be true for healthcare processes, nonprofits, as well as, for business and manufacturing. Manufacturing would begin with product design followed by production design to make it. Production design would not be guesswork. Production would be based on sound engineering principles to optimize production. Social services, human services, government operations, and healthcare programs should proceed on no less a foundation. Program theory represents the best that we know about these kinds of programs or production processes and is, therefore, instrumental in the design of a program; or at least could or should be. Nonetheless, the program must still be implemented based on some kind of thinking to achieve performance. Fidelity is the term applied to how closely program design for the program conforms to program theory’s determination of what has been shown to work best. Program theory, therefore, would be a logical starting point for finance and spending topics covered in a budget course. 2.2 Program Theory Program theory is one of the younger theories. Unlike astronomy, physics, chemistry or engineering which have long histories, program theory in building and evaluating public and nonprofit programs is relatively new dating from the late 1960s. The range of program theories has grown since then and can now be found in the literature of social science, public administration, psychology, criminal justice, education, and public health to name only a few. These disciplines have their individual disciplinary theories which their respective professionally degreed practitioners are expected to master. The practitioners are expected to apply those theories in therapy, social work outreach programs, policing, and teaching. Schwandt (2014) says program theory and program management can and should be mutually reinforcing and be used to solve problems in program operations. Debra Rog, who references Schwandt in her article “Infusing Theory into Practice, Practice into Theory: Small Wins and Big Gains for Evaluation”, supports Schwandt but goes on to point out “there remains a lack of clarity, specification, and systemization for much of what we do” (Rog, 2015, p. 224). Even though she writes in an evaluation context, programs are the primary element in that context. Both Schwandt and Rog are part of the teaching community of higher education. Programs are what budgets are created to operate. What is most often used in budgeting pertaining to program design and implementation is “common knowledge” or as Rog puts it, program design is approached with some kind of “technical” expertise built over time (Rog, 2015). Importantly, Rog goes on to draw on Bickman (2000) plus Bickman, Peterson (1990) and Lipsey (1993), defining program theory as “an underlying logic or theory of change for (program) evaluation, typically a program or an intervention” (Rog, 2015, p. 225). More simply put, program theory is the reasoning based on research of why a program can work. As Rog shows, program theory is most often not considered when evaluating a program; it certainly is not considered when designing a program. Many practitioners think program theory is for academics only. They, as a group, see no value in applying or even www.scholink.org/ojs/index.php/fce Frontiers of Contemporary Education Vol. 2, No. 2, 2021 45 Published by SCHOLINK INC. searching out program theory. In service learners are likely enough to bring such a mindset to a budgeting class. As a result of the contextual limitations described above, it is even more unlikely program theory is linked to budgeting for the program. A traditional budget built on line items in no way represents program work; only what is allotted for spending on its operation. Program design, let alone program theory, were not factors in creating budgetary assignment to a program’s appropriation; nor managing it nor evaluating it. Accordingly, it is imperative that program theory be introduced in a public administration curriculum as soon as the first course. Program theory should also be introduced early in other curricula for degrees in disciplines mentioned above. If not in an introductory course, then the budgeting course becomes the next best place since such a course would be open to the issue of performance. If program theory is reaffirmed as important in any program evaluation course, the foundation of content knowledge concerning program theory can become stronger. Nonetheless, program theory should be the underlying source for understanding program performance since simply examining line items, the most common budget format, does not allow for any performance assessment. Staying within spending limits of line items does not performance make. Failure to introduce and use program theory in a budgeting course creates an imbalance in what students learn in order to apply “the best that we know”. Students and teachers who guide them are in essence, victims of myopia about one of the key learning outcomes from an MPA or similar programs granting professional degrees. 2.3 Performance and Budgeting Alfred Tat-Kei Ho discusses Performance Budgeting Theory in his article, “From Performance Budgeting to Performance Management” noting the importance of linking performance and budgeting. The perspective he offers shows that over many years of effort urging use of performance budgeting there is little to show for it (Tat-Kei, 2018). Tat-Kei traces over 100 years of history to survey efforts to add performance to budgeting. He points to Schick’s “classic” theory of categorizing the control, management, and planning orientations as the first marker laid down toward a more systematic understanding of budgeting and performance. Tat-Kei also discusses a “more relaxed” form of Performance Budgeting Theory which he terms “performance informed budgeting”. This approach to adding performance to budgeting requires, he says, some form of “dialog theory”. Then there is “presentational performance budgeting” which shows decision makers performance information but Tat-Kei writes it is “rarely used” and is not even intended for decision makers during budget decision making. All of Tat-Kei’s points are valid and important. However, these approaches or theories are at least one step beyond the origination point of budgets (whether stated or not): the programs. www.scholink.org/ojs/index.php/fce Frontiers of Contemporary Education Vol. 2, No. 2, 2021 46 Published by SCHOLINK INC. Programs are the first step to improved budgeting and budget decision makers. While it is unquestionably true, as Tat-Kei states, important precursors must exist for performance. Those precursors include capability to think about performance; leadership for building consensus on its use; interest in performance in applying performance to budgeting; and a number of “layers” of challenges surrounding the budgeting task including staff competency about measurement information plus technology to use it. He does not discuss program theory as one of the precursors, but it must be considered in the list, perhaps even the primary precursor without which performance data may be off target more than it otherwise might be. Certainly, the challenges to performance criteria and program theory’s use are significant. Nonetheless, programs are at the base of budgeting. There would be no need for budgeting if programs did not exist. Given this point of view, the work of Campbell and Lambright is an illustration of program theory informing program performance (Campbell & Lambright, 2016). They set out to examine nonprofit program performance and its origination point. They apply program theory called Multiple Constituency Theory. Before doing so, they review program theories which frame performance as accountability writing. They make it clear that such approaches are imposed by funders rather than being applied by decision makers in budgeting. While certainly valuable for funding originators and are not to be abandoned, Campbell and Lambright argue for an expanded consideration of program performance. Their research points to the need for an enlarged definition of performance. Which means, in turn, a change in program designs to match the expanded performance criteria. Such a change would make program design from this perspective stronger. Campbell and Lambright offer yet another reason to include program theory is a graduate budgeting course. Persuading budget decision makers about the importance, value, and improvements program theory can make, must be a “selling job” to decision makers. Professional degree holders having taken budgeting and finance courses should be taught about the value of using program theory to inform performance, however. Yet, trying to sell decision makers on performance becomes even more difficult if the performance of the program is not up to standards based on performance measurement. Here it is important to emphasize measurement is not counting. Measurement requires a scale; counting does not. This distinction must be made unequivocal to graduate students so that they do not fall into the trap of using counts as performance gauges. Equally important for understanding measurement for performance is the situation in which program design was inadequate to the task to begin with and becomes based on counting, as in traditional rating performance. It is difficult to imagine production and performance so distant from the thinking of those charged with managing government programs, nonprofit enterprises, or healthcare. Even more difficult to explain is having to be persuaded to apply theory or principles or even best practices. Using the terms production, performance, and theory yielded almost 27,000 results from the author’s university library search engines. The results come from agriculture, water treatment, computer chip manufacturing, education, www.scholink.org/ojs/index.php/fce Frontiers of Contemporary Education Vol. 2, No. 2, 2021 47 Published by SCHOLINK INC. education policy, public speaking, and government enterprise analysis, among others. It should be pointed out here these results are just from the years 2017 to 2020, making them recent as well as relevant. The point here is not to criticize Tat-Kei but instead to agree with him and illustrate how the task of persuading budget decision makers on the value of performance budgeting could be accomplished. Program theory should precede performance budgeting not replace it. Nor is the intention here to ignore the barriers and challenges in moving performance budgeting ahead through the “layers” of challenges. The intention, instead, is to build a strong foundation of program design, development, implementation, and evaluation to allow for performance measurement information to be valuable and therefore “sellable”. Putting these elements of performance together in this sequence would build a learning pathway for students to take with them to employ on the job. 3. Research Methods A critical review of relevant conceptualizations found in the literature was undertaken to develop an exemplar course applying computational thinking bolstered by program theory, program performance, budgeting, and course pedagogy. The material found most relevant is considered below. 3.1 Data Sources and Analysis Program theory is not likely to be the first thing on the minds of decision makers, particularly the elected ones or funders in the nonprofit context. Program theory, however, is in the purview of those trained as MPAs or MBAs. MPAs are more apt to be at least familiar with the concept from their courses and seminars. O’Brien (1987), as cited in Clement and Bigby (2011), emphasizes program theory as the basis of performance. Without program theory they write “an organization is not able to assess how well they (sic) are implementing processes that are linked to expected outcomes”. Measuring workload, time on task, number of clients served, and the like which are counts, makes it all but impossible to understand or to determine why or, even if, outcomes did occur (assuming such outcomes are the appropriate ones for the processes used). Furthermore, if the appropriate outcomes did not occur, the counting numbers will not be the method of discovering why not. Since budgeting should be based on performance, repeating, even a supposedly successful program, would be guesswork without knowing about the link between work and results. Furthermore, not knowing what was to intended to work and did not, converts into no better than puzzlement. Budget course learners must be aware of these concepts. 3.2 Logic Models Another understanding of program theory and one which may be clearer to decision makers is logic modeling. A logic model is an illustration of work processes which, with careful attention, can be used as a diagram for viewing a program and its processes. Logic models, in other words, allow visualization of work process for the program. Visual representations offer the benefit of a way to see how the program processes are aligned and how those processes are supposed to link to results. www.scholink.org/ojs/index.php/fce Frontiers of Contemporary Education Vol. 2, No. 2, 2021 48 Published by SCHOLINK INC. Logic model visuals are also important for seeing how a program’s work process is supposed to align. Visuals provided an added benefit of becoming a “selling” mechanism to decision makers. An added value from such visuals would be the ability to determine the implementation fidelity with logic models and/or program theory. Munter et al. (2014), show the way on this issue in their article “Assessing Fidelity of Implementation Unprescribed, Diagnostic Mathematics Intervention”. Following Dusenbury et al. (2003), fidelity of implementation is implementing programs as intended. They also show examples of fidelity assessment criteria such as quality of program delivery, adherence, program participants’ responsiveness to the program, and program duration. Munter, et al., also indicate the benefit of understanding the programs “mechanisms for change” can be better assessed for refining program theory and improving understanding why a program works or does not. In so doing, a form of program evaluation is being created. Program evaluation can then become linked with budgeting for the program. The literature in public administration is replete with logic models. Program evaluation literature is one of the key areas where logic models can be found. An MPA must become aware of such literature and the value of logic models. 3.3 Program Theory, Logic Models, and Lean Management Douglas Martin (2017) points to using lean management connected to decision making based on program processes illustrations. He points to the linkages of these elements for addressing the “layers” of decision making around budgets in his examination of the civil service in the British context. He concludes that lean management applied to Social Security reduced the complexity of what had become unnecessarily bureaucratic generating layers of rules and decision making for the eligibility of claimants. Furthermore, lean generated “greater emphasis” on performance targets. It led to consolidation of what had been separate decision-making processes. Martin reports further “Lean was central to addressing more efficient and quicker ways of dealing with these areas of work” (Martin, 2017, p. 155). Lean Thinking and Management courses appear in MPA and MBA curricula. The lessons from lean coursework can be transferred directly to budget course material to train MPAs or MBAs in the decision-making portions of a budget course project. Value Stream Maps used in lean management represent both the present state of the process of program operations and its future state forming, in a real sense, a logic model also. The value stream may provide more detail in many cases, but the starting point and end point are the same as a logic model. An illustration of the use of value stream maps in this sense was research performed by Hurley et al. (2017). In the study they designed, the research was intended to understand the best patient pathways in national patient care used in Ireland. Using value stream mapping, the researchers examined protocols or standard operating procedures to find patient care operations that “promote effective patient flow through the continuum of care from presentation at the emergency department through safe discharge home” (Hurley et al., p. 2). The data from the study was to show the national Irish agency in charge of quality of care how the lean processes can provide what is needed to make care better. In other words, www.scholink.org/ojs/index.php/fce Frontiers of Contemporary Education Vol. 2, No. 2, 2021 49 Published by SCHOLINK INC. the value stream maps were to be used for decision making. The researchers even applied program theory to the approach they described. Lean thinking and management are capacity building elements for budgeting purposes also. As can be seen from this discussion, program theory, lean thinking and management, logic models, and dollars and cents can be construed as intertwined. The nexus of these concepts is therefore directly linked to budgeting and as such can be a part of a budgeting course. The program theory-based budget course can be delivered in a measured fashion to build toward the final element of an actual budget creation exercise. The concept of computational thinking becomes extremely germane here as part of the program theory section. It is the foundational element for building a course using the concepts needing linkage with budgeting discussed above. Uzumcu and Bay (2021) discuss conceptual thinking as a concept in teaching students 21 st century problem solving; in other words, courses like budgeting combining program theory, performance, and lean management principles. They write that computational thinking is being applied to a “wide range” of curricula internationally (p. 567). Citing Wing from 2010, Lee and Malyn-Smith, report that computational thinking is, in the first instance, reasoning in a way that formulates a problem and then devising its solution. These two steps allow “an information processing agent” to apply the solution to the problem (Wing, 2010, p. 1; Lee & Malyn-Smith, n.d., p. 9). Such a description clearly fits anyone involved in program management and budgeting for that program. Hadad et al. (2021), build on this premise and its consequences by drawing on Zviel-Girshin et al., writing “Professionals in the knowledge society need digital skills suitable for coping with constantly evolving Information and Communication Technologies (ICT), digital tools, and applications (Hadad et al., p. 764; Zviel-Girshin et al., 2020). The budgeting process would be well served by having educated personnel who have the first order element of these skills: computational thinking. 3.4 Computational Thinking Computational thinking is also termed procedural thinking (Papert, 1996; in Cansu & Cansu, 2019). For teaching budgeting procedural thinking might be less intimidating daunting for learners who think themselves less than proficient in mathematics or even arithmetic. Procedural thinking is a stepwise construction which can create a logic model and, therefore, a visual toward algorithmic thinking. Computational thinking overlaps with procedural thinking since both seek a logic toward finding a problem solution. Equally important, logic modelling for computational/procedural thinking is a direct pedagogically, sensible way to introduce program theory as logic modelling. The pedagogical approach using logic models would be a form of creating a learning context as described by Allen and Heredia (2021). Contexts can be limiting factors, as they explain them. Naturally, then reducing the influence of learning limitations would be important to integrating program theory into a budget course or, any analytic course for that matter. One way to do so, would be what Heredia and Tan (2021) did in www.scholink.org/ojs/index.php/fce Frontiers of Contemporary Education Vol. 2, No. 2, 2021 50 Published by SCHOLINK INC. working with a graduate course: create a context for learning using active learning or learning by doing. Learning by doing is another way of saying learning by making as discussed below. Since computational thinking or procedural thinking both involve seeking problem solutions, it is valid to apply them to budgeting and a budgeting course. Existing budget numbers or expense numbers are most often defined as the problem rather than program theory or program design. Dollars for line items representing a program forms the problem because of limitations of revenue, time, and the imperative to show spending as the important issue. Papert writes, as cited in Cansu and Cansu, that procedural thinking can be interpreted as individual thinking about problem solving and involves “learning by making” (Papert, 1991, p. 1; in Cansu & Cansu, p. 1). A budget course would provide just such learning by making with a budget creation assignment, but one that also included program theory and program design resulting from that theory. A number of elements of computational thinking or procedural thinking that are logically associated with learning by making and would be useful in a budget course are  Formulating  Analyzing and dividing problems into manageable constituent parts  Organizing and modelling  Efficiency  Critical Thinking and creativity  Data Analysis All these features of computational thinking would be needed in making a course budget project. The very important element tying all of these facets of computational thinking together is a set of systematic logical steps making a replicable solution, i.e., an algorithmic way of thinking which is a form of synthesis. In this sense it “is the process of constructing a scheme of ordered steps which may be followed to provide solutions to all constituent problems necessary to solve the original problem” (Cansu & Cansu, p. 4). The synthesis step is found in the program and performance-based budget activity. Others of the list above are usually part of a budgeting course in one way or another. The pedagogical aim of using procedural learning in a budget course is putting these analytic processes in a logical order and using them to create a program and performance-based budget assignment. In their article on using computational thinking in an education setting, Kang and Lee (2020) call the approach described above as “project learning” and is found “in every discipline in schools for many years” (Kang & Lee, p. 5358). They add the approach has been “very popular”. They cite a large number of examples of successfully using computational or procedural thinking in courses (Kang & Lee, pp. 5360-5361). Kang and Lee also add that the approach of procedural or computation thinking has appeared in courses found in liberal arts (Kang & Lee, p. 5361). Caldron provides analysis for a more traditional classroom exercise intended to have students work in role plays and problem-solving exercise aimed at computational thinking. This teaching approach is not project learning, but it is learning by doing. The pretest and posttest results reported by Caldron showed www.scholink.org/ojs/index.php/fce Frontiers of Contemporary Education Vol. 2, No. 2, 2021 51 Published by SCHOLINK INC. “a positive correlation between teaching abstraction and an enhancement in learner’s ability to engage in abstract thinking; and a positive correlation teaching algorithmic thinking and an enhancement in sequentiality” (Caldron, pp. 1190-1191). The Caldron role play results show formation of the two elemental features of computational thinking: abstract thinking and sequentiality. Abstract thinking allows a person to recognize patterns, spot then eliminate irrelevant facets of a problem, followed by generating a framework for diagnosing similar problems (Calderon, 2020). Sequentiality involves using a step-by-step process in an unambiguous fashion. There is no pedagogical reason that project learning or role plays and problem solving could not be combined. In a budget course, for example, role plays of stakeholders in the budgeting process could be designed followed by a project such as preparing a budget using hypothetical budget numbers and programs. The assignment could be to find the least expensive supplier of a key expense such as an item of equipment. Problem solving exercises in which the assignment is to work through a budget problem such as determine the best combination of resources for a hypothetical nonprofit theater deciding on how to balance marketing/outreach with the need to add new stage lighting. Suitable variations could be made for a government operation or health care organization. Such exercises in the language of teaching computational thinking are “unplugged”. That is to say, they do not necessarily need computer applications for calculations. In fact, there may be no actual mathematical work needed. An Example of one such unplugged assignment was developed and evaluated for learning outcomes by Mitri et al. (Mitri et al., 2017). The authors describe their work using role plays as active learning devising role plays which encourage, they say, participation and engagement. This description sounds remarkably similar to that described by Calderon et al. The similarities begin with the assignments being unplugged. Another similarity is learning by doing. Yet another similarity is following a sequence of steps. The desired learning outcomes involved “understanding” and “relating” information systems to strategy. While the term computational thinking is never used in the study, clearly the same kinds of cognitive skills would need to be applied as the Calderon, et.al, research highlighted. Furthermore, the role play exercises and learning outcomes included the subject of budgeting in the course they studied. 4. Discussion While cognitive thinking emerged from Computer Science it need not be used exclusively for computer programming, as was its original purpose. A major use of computational thinking is problem solving based on mental steps to discern patterns, disassemble them, and then organize them for a solution to a problem. The need for analytic problem solvers is critical in many fields and disciplines. While computational thinking can yield an organized sequence of elements to solve a problem, not all problem solvers need to be able to algorithmic thinking. Wing noted in her definition of the concept the end point would be www.scholink.org/ojs/index.php/fce Frontiers of Contemporary Education Vol. 2, No. 2, 2021 52 Published by SCHOLINK INC. the information processing agent to make the solution operational. The information processing agent can be other than the problem solver. In any case, the solution must be applied to the problem by someone competent in the analytics found in computational thinking. MPAs and MBAs would be the target audience for such thinking. Those information processing agents form the drivers of such fields as informatics and its full array of disciplines. Informationalists in informatics are trained to use databases to extract information for examination and use by decision and/or policy makers. The education of these groups includes graduate degrees such as MPA, MBA, Health Administration, and nonprofits. Analytically oriented courses, such as budgeting or finance or program evaluation, in these degrees present the opportunities illuminate the importance of computational thinking drawing on program theory, logic modelling, value steam mapping, and performance. As a result, the future decision makers or policy makers would be up to date on what their professional future is likely to present as problems and the way to solve them. Based on the research literature presented and reviewed here, the task of teaching the subject of computational thinking can be done for efficacy. While the focus here was a budgeting course, as was seen in the early sections above, the courses to which it can be applied are wide and diverse. References Allen, C. D., & Heredia, S. C. (2021). Reframing organizational contexts from barriers to levers for teacher learning in science education reform. Journal of Science Teacher Education, 32(2), 148-166. https://doi.org/10.1080/1046560X.2020.1794292 Bickman, L. (2000). Summing up program theory in evaluation. New directions for program evaluation, 87, 103-112. https://doi.org/10.1002/ev.1186 Bickman, L., & Peterson, K. A. (1990). Using program theory to describe and measure program. New Directions for Program Evaluation, 47, 61-72. https://doi.org/10.1002/ev.1555 Calderon, A. C., Skillicorn, D., Watt, A., & Perham, N. (2020). A double dissociative study into the effectiveness of computational thinking. Education and Information Technologies, 25(2), 1181-1192. https://doi.org/10.1007/s10639-019-09991-3 Campbell, D. A., & Lambright, K. T. (2016). Program performance and multiple constituency theory. Nonprofit and Voluntary Sector Quarterly, 45(1), 150-157. https://doi.org/10.1177/0899764014564578 Cansu, S. K., & Cansu, F. K. (2019). An overview of computational thinking. International Journal of Computer Science Education in Schools, 3(1), 1-11. https://doi.org/10.21585/ijcses.v3i1.53 Clement, T., & Bigby, C. (2011). The development and utility of a program theory: Lessons from an evaluation of a reputed exemplary residential support service for adults with intellectual disability and severe challenging behaviour in Victoria, Australia. Journal of Applied Research and Intellectual Disabilities, 24(6), 554-565. https://doi.org/10.1111/j.1468-3148.2011.00634.x https://doi.org/10.1080/1046560X.2020.1794292 https://doi.org/10.1002/ev.1186 https://doi.org/10.1002/ev.1555 https://doi.org/10.1007/s10639-019-09991-3 https://doi.org/10.1177/0899764014564578 https://doi.org/10.21585/ijcses.v3i1.53 https://doi.org/10.1111/j.1468-3148.2011.00634.x www.scholink.org/ojs/index.php/fce Frontiers of Contemporary Education Vol. 2, No. 2, 2021 53 Published by SCHOLINK INC. Dusenbury, L., Brannigan, R., Falco, M., & Hansen, W. B. (2003). A review of research on fidelity of implementation: Implications for drug abuse prevention in school settings. Health Education Research, 18(2), 237-256. https://doi.org/10.1093/her/18.2.237 Hadad, S., Shamir-Inbal, T., Blau, I., & Leykin, E. (2021). Professional development of code and robotics teachers through small private online course (SPOC): Teacher centrality and pedagogical strategies for developing computational thinking of students. Journal of Educational Computing Research, 59(4), 763-791. https://doi.org/10.1177/0735633120973432 Heredia, S. C., & Tan, E. (2021). Teaching and learning in makerspaces: Equipping teachers to become justice-oriented maker-educators. Journal of Educational Research, 114(2), 171-182. https://doi.org/10.1080/00220671.2020.1860871 Howard, F. (2002). Teaching performance measurement in a core MPA budgeting class. Journal of Public Affairs Education, 8(2), 141-150. https://doi.org/10.1080/15236803.2002.12023543 Hurley, E. M., McHugh, S., Carroll, A., & White, B. (2017). A protocol to evaluate the effectiveness of a nationwide acute medicine programme: Examining what works, for whom, how, and in what circumstances. International Journal of Integrated Care (IJIC), 17(Supplement), 1-2. https://doi.org/10.5334/ijic.3519 Kang, Y., & Lee, K. (2020). Designing technology entrepreneurship education using computational thinking. Education and Information Technologies, 25(6), 5357-5377. https://doi.org/10.1007/s10639-020-10231-2 Lee, I., & Malyn-Smith, J. (2020). Computational thinking integration patterns along the framework defining computational thinking from a disciplinary perspective. Journal of Science Education and Technology, 29(1), 9-18. https://doi.org/10.1007/s10956-019-09802-x Lipsey, M. W. (1993). Theory as method. New Directions in for program evaluation, 57, 5-38. https://doi.org/10.1002/ev.1637 Martin, D. (2017). Making tax and social security decisions: Lean and deskilling in the UK civil service. New Technology, Work, & Employment, 32(2), 146-159. https://doi.org/10.1111/ntwe.12092 Millar, R., Morton, A., Bufali, M. V., Engels, S., Dabak, S. V., Isaranuwatchai, W., Chalkidou, K., & Teeraawattananon, Y. (2021). Assessing the performance of health technology assessment (HTA) agencies: Developing a multi-country, multi-stakeholder, and multi-dimensional framework to explore mechanisms of impact. Effectiveness & Resource Allocation, 19(1), 1-14. https://doi.org/10.1186/s12962-021-00290-8 Mitri, M., Cole, C., & Atkins, L. (2017). Teaching case: a systems analysis role-play exercise and assignment. Journal of Information Systems Education, 28(1), 1-9. Munter, C., Wilhelm, A. G., Cobb, P., & Corday, D. S. (2014). Assessing fidelity of implementation of an unprescribed, diagnostic Mathematics Intervention. Journal of Research on Educational Effectiveness, 7(1), 83-113. https://doi.org/10.1080/19345747.2013.809177 https://doi.org/10.1093/her/18.2.237 https://doi.org/10.1177/0735633120973432 https://doi.org/10.1080/00220671.2020.1860871 https://doi.org/10.1080/15236803.2002.12023543 https://doi.org/10.5334/ijic.3519 https://doi.org/10.1007/s10639-020-10231-2 https://doi.org/10.1007/s10956-019-09802-x https://doi.org/10.1002/ev.1637 https://doi.org/10.1111/ntwe.12092 https://doi.org/10.1186/s12962-021-00290-8 https://doi.org/10.1080/19345747.2013.809177 www.scholink.org/ojs/index.php/fce Frontiers of Contemporary Education Vol. 2, No. 2, 2021 54 Published by SCHOLINK INC. O’Brien, J. (1987). A Guide to Life-Style Planning: Using the Activities Catalog to Integrate Services and Natural Support Systems. In B. Wilcox, & G. Bellamy (Eds.), The Activities Catalogue: An Alternative Curriculum for Youth and Adults with Severe Disabilities. Brooks: Baltimore. Papert, S., & Harel, I. (1991). Situating constructionism. In S. Papert, & I. Harel (Eds.), Constructionism. New York: Ablex Publishing. Peters, B. G. (2019). American public policy: promise and performance (11th ed.). Thousand Oaks: Sage. Pope, A. M., Finney, S. J., & Bare, A. K. (2019). The essential role of program theory: fostering theory-driven practice and high-quality outcomes assessment in student affairs. Research & Practice in Assessment, 14, 5-17. Rog, D. J. (2015). Infusing Theory into Practice, Practice into Theory: Small Wins and Big Gains for Evaluation. American Journal of Evaluation, 36(2), 223-238. https://doi.org/10.1177/1098214015573068 Schwandt, T. A. (2014). On the mutually informing relationship between practice and theory in evaluation. American Journal of Evaluation, 35, 232-236. https://doi.org/10.1177/1098214013503703 Tat-Kei, A. (2018). From performance budgeting to performance budget management: Theory and practice. Public Administration Review, 78(5), 748-758. https://doi.org/10.1111/puar.12915 Uzumcu, O., & Bay, E. (2021). The effect of computational thinking skill program design developed according to interest driven creator theory on prospective teachers. Education and Information Technologies, 26(1), 565-583. https://doi.org/10.1007/s10639-020-10268-3 Wing, J. (2010). Computational thinking: What and why? Pittsburgh, PA: Carnegie Mellon University, School of Computer Science. Wing, J. M. (2006). Computational thinking. Communications of the ACM, 49(3), 33-35. https://doi.org/10.1145/1118178.1118215 Zviel-Girshin, R., Luria, A., & Shaham, C. (2020). Robotics as a tool to enhance technological thinking in early childhood. Journal of Science Education and Technology, 29(2), 294-302. https://doi.org/10.1007/s10956-020-09815-x https://doi.org/10.1177/1098214015573068 https://doi.org/10.1177/1098214013503703 https://doi.org/10.1111/puar.12915 https://doi.org/10.1007/s10639-020-10268-3 https://doi.org/10.1145/1118178.1118215 https://doi.org/10.1007/s10956-020-09815-x