Jai | Journal of Accounting and Investment, vol. 20 no. 1, January 2019 Evaluation of E-Budgeting Implementation in Provincial Government of DKI Jakarta Using CIPP Model Approach Rama Andika Thio Rahman*, Gugus Irianto, Rosidi ABSTRACT: This research aims at analyzing the effectiveness of e-Budgeting implementation with the Provincial Administration of Special Capital Region of Jakarta using the CIPP (Context, Input, Process and Product) model of Stufflebeam (2003). This qualitative research employs a case study approach and collects the data through interview and documentation. The Informants of this research are the Local Government Budget Team (TAPD) of the Special Capital Region of Jakarta Province. The results of this research show that the e-Budgeting implementation with the Provincial Administration of Special Capital Region of Jakarta is worthy to be continued since it evidently runs effectively with regard to the four components of CIPP Model. With regard to Context, it focuses on successful achievement of initial purposes, which are budget transparency and accountability. The finding of context evaluation is a shift of users’ paradigm from old system to e-Budgeting. With regard to Input, it uses adequate facilities and infrastructures as well as human resources, and the results of input evaluation show that an improvement of network and server used is necessary. With regard to Process, it facilitates and minimizes error in RKA arrangement process. The results of process evaluation show that there is no legally applicable SOP in e-budgeting implementation stages. With regard to output (Product), it improves Local Budget quality in case of its appropriateness to RPJMD planning document, and active role of budget control management. KEYWORDS: evaluation; e-budgeting; CIPP; budgeting; provincial government Introduction The background of this research is the enactment of law No.14 year 2008 on the openness of delivering public information. This law enactment substantially provides a constitutional security that any practice of democracy and good governance are meaningful to policy making process related to public interest. To realize good governance, particularly with regard to the accountability and transparency of public information, it requires directed policy measures to change an institutional and governance system through utilization of ICT (Information and Communication Technologies) (Sari & Winarno, 2012). Furthermore, a National Policy and Strategy of e-Government Development is issued through Presidential Instruction of the Republic Indonesia Number 3 Year 2003, stating that every Governor and Regent/Mayor is mandated to take necessary measures pursuant to their respective duties, functions and authorities for national implementation of e-Government development. E- government has rapidly developed for the last 15 years, in which the United Nation Development Program (UNDP) measures the E-Government Development Index (EGDI) ranging from 0.25 to 1.00. The Indonesian Department of Accounting, Universitas Brawijaya, Jalan Veteran, Ketawanggede, Lowokwaru, Kota Malang, East Java 65145, Indonesia. *Correspondence: ramaandikathio@gmail.com This article is avalilable in: http://journal.umy.ac.id/index.php/ai DOI: 10.18196/jai.2001110 Citation: Rahman, R. A. T., Irianto, G., & Rosidi. (2019). Evaluation of E- Budgeting Implementation in Provincial Government of DKI Jakarta Using CIPP Model Approach. Journal of Accounting and Investment, 20(1), 94-114. Received: 10 September 2018 Reviewed: 11 September 2018 Revised: 19 September 2018 Accepted: 04 October 2018 Copyright: © 2019 Rahman, Irianto & Rosidi. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. mailto:ramaandikathio@gmail.com http://journal.umy.ac.id/index.php/ai http://journal.umy.ac.id/index.php/ai/article/view/5026 http://journal.umy.ac.id/index.php/ai/index Rahman, Irianto and Rosidi / Evaluation of E-Budgeting Implementation in Provincial Government of DKI Jakarta… Journal of Accounting and Investment, January 2019 | 95 E-Government Development Index (EGDI) position ranges from 0.25 – 0.50 (middle-EGDI). This value range is far below the EGDI value of other Asian nations, such as Singapore, Japan and Israel with EGDI value ranging from 0.75 – 1.00 (high-EDGI) (Survey, 2016). Based on the survey results, we may state that the e-Government system development in Indonesia is not maximal yet. Non-maximal information system users in a governmental management will lead to less effective services to the public. Besides, possible mistakes in the managing the government’s resources, particularly financial management process will increase. E-Budgeting is one embodiment of e-Government implementation in government’s financial management. E-Budgeting is a system developed to accelerate government’s planning and budgeting process. E-Budgeting is a system created to prepare the source of standard unit price data as mandatory guideline for budget arrangement, arrange activity proposal as well as its details for various working units, facilitate internal discussion or external discussion with DPRD until revision stage with the government and DPRD to reach a mutual agreement in budgeting (Prasetyo, 2006). In Indonesia, the regional governments which have applied the e-Budgeting system include Surabaya City and the Special Capital Region of Jakarta Province. The phenomena related to the Regional Budget fund of the Jakarta Provincial Administration include the case of procurement of garbage truck of the Sanitary Agency. In 2014, sanitary truck procurement is not performed through tender, but through purchase with e-catalog. However, when it is time to pay to the holder of vehicles, no payment can be made for the trucks because of shortage of budget with the Sanitary Agency for Rp 11 Billion (Kementerian Koordinator Bidang Perekonomian Republik Indonesia, 2015). Although the e-Budgeting system has been used, however, in the case above, we may assume that the system has not been utilized optimally. There are various constraints related to e-Budgeting implementation with the Provincial Administration of Special Capital Region of Jakarta, including non-optimal computerization process and that the human resources to manage and execute it have not fulfilled the required competency. In this case, more trainings are required for the human resources of the Provincial Administration of Special Capital Region of Jakarta in order to enhance their ability to perform arrangement with the system (Kementerian Koordinator Bidang Perekonomian Republik Indonesia, 2015). According to the phenomena and constraints faced in e-budgeting system implementation, an evaluation needs to be conducted. E-budgeting is a system which has been implemented, thus it needs to be assessed or evaluated for its usage in order to examine to what extent its achieves the designated purposes and objectives (Weber, 1999). The information system evaluation may be viewed from two perspectives, which are efficiency and effectiveness. Efficiency emphasizes on system availability quality, including, for example, system reliability, total access time, total downtime, etc. Meanwhile, effectiveness reviews the information system from quality perspective as compared with the expectation and reality of the system (Weber, 1999). Previous research related to e-Budgeting implementation has been conducted by Purwohandoko, Sanaji, & Mustofa, (2015) using the Technology Acceptance Model (TAM) approach, of which results propose Rahman, Irianto and Rosidi / Evaluation of E-Budgeting Implementation in Provincial Government of DKI Jakarta… Journal of Accounting and Investment, January 2019 | 96 that the role of a manager in its implementation stage may build user’s trust. The next determining factor is experience in using such system and user’s satisfaction in operating the e-Budgeting system. Another research which focuses on the e-Budgeting implementation is the research conducted by Sangadji (2015), which describes the Critical Success Factor (CSF) of e- Budgeting implementation with the administration of Surabaya City, of which results explain that the main factors of e-Budgeting successful implementation with the Administration of Surabaya City are support and commitment of the Top Management. The previous researches, as explained above, show that there are various evaluation frameworks which focus on the factors influencing the e- Budgeting implementation, but with less discussion in the perspective of effectiveness. To measure the effectiveness of a system is important since it is to examine to what extent the system achieve the purposes and objectives (Stair & Reynolds, 2010). Local governments spend much cost to create a system and to apply it. Therefore, local governments pay great attention to the system’s performance. E-budgeting is a system which has been implemented, thus it needs to be assessed and evaluation for its usage in order to examine how successful it achieves its designated purposes and objectives (Weber, 1999). The evaluation in this research is conducted using the CIPP (Context, Input, Process and Product) model (Stufflebeam, 2003). The CIPP model approach is used since it is oriented to formative and summative evaluation (Quezada, 2005). Formative evaluation is used to improve a program as long as the program is ongoing by providing information of how well the program runs. Through this formative evaluation, any inefficiency may be detected in order to make improvement. Meanwhile, summative evaluation aims at measuring the effectiveness of whole program to decide whether the program currently implemented is worthy for continuance and application, and then for redevelopment (Farsi & Sharif, 2014). In addition, this evaluation model also aims at connecting information, context, input and process to product, and to show that environment is available for achieving the purposes and objectives of implemented program (Yahaya, 2001). Until now, only few literatures try to apply the CIPP evaluation model approach, that this approach is deemed relatively representative to extract the implementation of a program. The results of research conducted by (Hasan, Yasin, & Yunus, 2015) propose that the CIPP evaluation model may describe the effectiveness of formative and summative points of view and has the ability to solve problems faced in the implementation stage of a program. Based on this condition, a research is conducted focusing on the framework of CIPP model with a purpose to conclude an assisting framework in evaluating e-Budgeting system implementation. The assisting framework resulted is then applied to evaluate the e-Budgeting program with the Provincial Administration of Special Capital Region of Jakarta. According to the background above, the purpose of this research is to analyze the effectiveness of e-Budgeting program implementation with the Provincial Administration of Special Capital Region of Jakarta as reviewed from the four points of view of CIPP model, which are Context, Input, Process and Product. Rahman, Irianto and Rosidi / Evaluation of E-Budgeting Implementation in Provincial Government of DKI Jakarta… Journal of Accounting and Investment, January 2019 | 97 There are two aspects of research contribution in this study: theoretical contribution and practical contribution. Theoretically, this study provides empirical evidences of CIPP evaluation model (Context, Input, Process and Product) in e-Budgeting program implementation analysis, and proves that the CIPP evaluation model may measure the effectiveness of a program currently in operation. The practical contribution of this research may be applied to the e-Budgeting planning and implementation stages with all provinces and regions throughout Indonesia. In addition, this study provides information for consideration and learning to other local governments in implementing an e-Budgeting policy as part of e-Government. In addition, with more thorough knowledge of e-budgeting, it is expected to open local government agencies’ insight to consider various important elements in information system development process to be used in budgeting process. Literature Review and Research Focus Electronic Government (E-Government) Public sector’s performance improvement requires the government to use information and communication technology in the course of public sector reformation. Information and communication technology in government sector is a tool used to operate governmental process. Information and communication technology used in operating the governmental process may also improve the government’s good governance (Sangadji, 2015). Information and communication technology used in the public sector is called electronic government or e-Government. According to Presidential Decree No. 20 Year 2006, e-government is the utilization of information and communication technology in governmental process to improve the efficiency, effectiveness, transparency and accountability of government administration. According to the United Nation Development Programme (UNDP), “electronic government is the application of information and communication technology (ICT) by government agencies”. Both definitions of e-Government above show that e-government is information and communication technology used and utilized in the implementation process of government’s activities to achieve its purposes and objectives and to improve the relationship between the government and other parties. This information technology usage then results in new form of relationship. According to Yildiz (2007), there are some relationship models of e- Government as shown in Table 1. Electronic Budgeting (E-Budgeting) E-Budgeting is a budgeting system which contains web-based computer program application to facilitate local budgeting process. E-budgeting starts when the Administration of Surabaya City is aware that in the old budgeting, the unit prices of expenditure items are unstandardized, thus misappropriation may occur, and it is evident that the Executive Budgeting Team hardly control meeting and memorize in detail of the proposal of Local Government Agency (SKPD), especially that SKPD does not attend the meeting. The purpose is to facilitate the Executive Budgeting Team to arrange the budget and improve the quality of Regional Budget (APBD) from Rahman, Irianto and Rosidi / Evaluation of E-Budgeting Implementation in Provincial Government of DKI Jakarta… Journal of Accounting and Investment, January 2019 | 98 Table 1 Electronic Government Models Parties of Communication Content Dominant Characteristics Definition Example Government-to- Government (G2G) Government Information and service Communication, coordination, standardization of information and service e-adminstration Establishing and using a common data warehouse Government-to- Citizen (G2C) Communication, transparency, acoountability, effectiveness, efficiency, standardization of information and services, productivity. e-government Government organization websites, e-mail communication between the citizens and government officilas. Government-to- Business (G2B) Communication, collaboration, commerce e-government, e-Commerce, e-Collaboration Posting government bids on the web, e-procurement, e-partnerships. Government-to- Civil Society (G2SC) Communication, coordination, transparency, accountability. e-government Electronic communication and coordination efforts after a disaster Citizen-to-Citizen (C2C) Communication, Coordination, transparancy, accountability, grassroots organization. e-government Electronic discussion groups on civic issues Source: Yildiz (2007) the perspective of appropriateness to the Regional Medium-Term Development Plan (RPJMD), the accuracy of value and account and the accountability of expenditure allocation. This system has the following benefits: 1. Transparent process, 2. Short and timely adjustment process, 3. Budget is as needed for activity, 4. Clear budget chronology, and 5. Report as requested by Head of Budgeting Team may be fulfilled. The e-Budgeting system, like other e-Government programs, has advantages in work effectiveness and cost efficiency since many works may be completed by only a few people within a short time, and with the paperless culture in e-Budgeting, the government does not necessarily provide lots of paper to all SKPDs to fill in the forms, thus it is cost efficient. E-Budgeting is an information system used in government’s budgeting process. E-Budgeting is used in budgeting process as a means of facilitating collection of standard unit price data (particularly for direct expenditure components). Therefore, an understanding of manual APBD arrangement process is necessary, and thus we will be able to know the e-Budgeting implementation process in APBD arrangement process. Manual APBD arrangement process may help give an understanding and technical direction in APBD arrangement and utilization of e-budgeting system developed by the Provincial Administration Rahman, Irianto and Rosidi / Evaluation of E-Budgeting Implementation in Provincial Government of DKI Jakarta… Journal of Accounting and Investment, January 2019 | 99 of Special Capital Region of Jakarta in regional budget planning and arrangement process. Examining APBD arrangement process, we will be able to identify any parties involved in APBD arrangement and any parties using the e-budgeting in APBD arrangement process. Public Policy Evaluation A public policy evaluation may be understood as an evaluation of policy implementation (Winarno, as quoted by Dwidjowito (2004). According to Dunn (1999), the term evaluation may be equalized to appraisal, rating, and assessment. Evaluation is related to information of the value or benefit resulted from a policy. Evaluation provides valid and trusted information of the performance of a policy, which is to what extent the need for value and opportunity has been achieved through public action. Evaluation contributes to clarification and criticism of the values underlying the selection of purposes and targets and contributes to other policy analysis methods, including formulation of problem and recommendation. According to Wibawa, as quoted by Nugroho (2006), public policy evaluation has four functions, which are explanatory function, compliance function, audit function and accounting function. The four functions of evaluation are related to the appropriateness of evaluation target to the suitability of pattern and field application, thus the evaluator will be able to understand and solve problems which may arise. CIPP Evaluation Model This evaluation model is commonly known and applied by evaluators. The concept of CIPP evaluation model (Context, Input, Process and Product) is first introduced by Stufflebeam in 1965 as the results of his effort in ESEA (the Elementary and Secondary education Act) evaluation (Stufflebeam, 2003). Stufflebeam’s evaluation model consists of four dimensions, which are Context, Input, Process and Product, and is thus called CIPP. The four dimensions are the targets of evaluation, which are components and process of a program implemented. In comparison to other assessment model, the CIPP dimensions, which are Context, Input, Process and Product, may be a research instrument, with a purpose to facilitate and emphasize more on the focus to be developed and the information obtained may help in decision making. In addition, the information given may help in decision making of the continuance of a program for its betterment in the future with regard to planning, governance, innovation and modification of program currently in operation (Hasan et al., 2015). The advantage of CIPP evaluation model compared to other evaluation model is that using the CIPP model will result in an analysis on the strengths and weaknesses of program implemented, which aims at improving the quality of such program (Mohebbi, Akhlaghi, & Khoshgam, 2011). In the research conducted by Hakan & Seval (2011), using the CIPP evaluation model may assess to what extent a program is effectively implemented from the perspective of achievement of objectives, utilization of adequate resources, both human resources and information technology infrastructure, and output generated. Rahman, Irianto and Rosidi / Evaluation of E-Budgeting Implementation in Provincial Government of DKI Jakarta… Journal of Accounting and Investment, January 2019 | 100 Below is an explanation of various dimensions proposed by Darodjat & Wahyudhiana (2015). First, context evaluation, which is an evaluation which focuses on the status of object in overall, identifies weaknesses and strengths, diagnoses problems and provides their solutions, tests whether the purposes and priorities are adjusted to the needs to be implemented. Second, input evaluation, which is to determine how the purposes of a program are achieved. Input evaluation may help manage decision, determine existing sources, what alternative to take, what plan and strategy to achieve the purposes, and what working procedure to achieve them. The components of input evaluation are human resources, facilities and support tools, and various necessary procedures and rules. Third, process evaluation, which is used to predict procedure design or implementation design during implementation stage, provide information for program decision and serve to be recording or archive of last procedures. This process evaluation includes collection of assessment data determined in and applied to program implementing practice. Fourth, product evaluation, which is an assessment performed to measure how successful the designated purposes are achieved. The data generated will highly determines whether the program will be continued, modified or ceased. Based on the methods above, we may examine that each method has strengths and weaknesses. With regard to evaluation to be performed and based on the description of such evaluation theories, the CIPP method of Stufflebeam will be used. The considerations on which the determination is based are as follows: first, this method has clear and orderly measures in the explanation of each program sequence; second, the CIPP method may analyze in detail the planning, application, implementation and continuation processes; third, the CIPP model is a model known to and applied by evaluators more than any other models. Below are explanations of the four components of CIPP evaluation model: 1) Context Evaluation Arikunto (2014) explains that context evaluation, according to Stufflebeam, is used to answer the question “what we need to do?” This evaluation identifies and assesses any needs underlying the arrangement of program. For example, regulation underlying the formation of policy, purposes of the existence of a program and identification of problems related to the policy. Another opinion proposed by Mahmudi (2011) states that the purpose of context evaluation is to diagnose problems faced by organization and to assess whether the designated purposes and priorities have met the needs of the parties targeted by the organization. 2) Input Evaluation The second stage is input evaluation, which is an evaluation used to find the answer of the question “how we do it?” This evaluation identifies and assesses aspects which play a role in the management and success of program implemented. The results of such identification may become the basis of decision making of what measures and resources are needed to achieve the purposes and objectives of the program (Arikunto, 2014). According to Widyoko, as quoted by Sesmiarni (2014), input evaluation helps regulate decision and determine existing sources, what alternative to take, Rahman, Irianto and Rosidi / Evaluation of E-Budgeting Implementation in Provincial Government of DKI Jakarta… Journal of Accounting and Investment, January 2019 | 101 what plan and strategy to achieve the purposes and what working procedure to achieve them. The components of input evaluation include: first, human resources; second, supporting facilities and tools; third, various necessary procedures and rules. 3) Process Evaluation According to Arikunto (2014), the proses evaluation in CIPP model is directed to how far activities in the program have been implemented pursuant to the plan. Process evaluation helps identify implementation process, thus the result may show whether the process has been appropriately implemented with the existence of program, and helps correct program currently in operation (Quezada, 2005). Process evaluation is used to detect procedure design or implementation design and provides information for program decision and serves to be recording of past procedure. Basically, process evaluation is used to examine to what extent a plan has been applied and what components need to be corrected. 4) Product Evaluation According to Arikunto (2014), the final stage of CIPP model is product evaluation (output). This evaluation aims at answering the question “is the program successfully implemented?” Widyoko in Sesmiarni (2014) argues that product evaluation serves to be assisting tool to make further decision, both of achieved outcome and what to do after the program is operated. This evaluation illustrates the outputs and benefits obtained from a program implementation. In addition, the results of process evaluation are expected to help leaders in decision making regarding whether it is to continue, cease or modify the program. Previous Researches Table 2 depicts some previous researches regarding e-Budgeting. Based on previous researches related to e-Budgeting as part of e-Government, the differences of this research and previous researches are the research subject, approach model and research purpose. The subject of this research is the Local Government Budget Team the Special Capital Region of Jakarta Province. The research model used in this research is the CIPP model (Context, Input, Process and Product) (Stufflebeam, 2003). The purpose of this research is to analyze the effectiveness e-Budgeting implementation with the Provincial Administration of Special Capital Region of Jakarta using the CIPP model (Stufflebeam, 2003). Research Method This research employs a qualitative approach and a case study. The researcher employs a qualitative approach in order to examine information in detail and complete to what extent the effectiveness of e-Budgeting system with the Provincial Administration of Special Capital Region of Jakarta is. This information may only be obtained by directly talking to executors. This is in line with what is stated by (Creswell, 2014), that qualitative Rahman, Irianto and Rosidi / Evaluation of E-Budgeting Implementation in Provincial Government of DKI Jakarta… Journal of Accounting and Investment, January 2019 | 102 Table 2 Previous Researches No Researcher Research Purpose Results and Recommendation 1 Jones, Irani and Sharif (2007) To provide information of organization aspect to improve understanding of e- Government The results of this research propose that e- Government utilization in governmental affairs needs evaluation in its practice by improving knowledge and understanding of e- Government. 2 Chen and Zhang (2012) To measure the performance of e- Government in the society The results of this research emphasize more on public participation in using e-Government. It suggests that the implementation stage needs redevelopment and improvement 3 Anwar, Djumiarti and Rihandoyo (2013) To examine factors which inhibit and support e- government application. The results of this research propose that the main elements of success in e-Government implementation are government’s support, ability to operate it and benefit of the application. This research suggests making of support regulation of e-Government and re- improvement of human resources (HRs). 4 Purwohandoko et al. (2015) To identify the relationship between organization support, supervision support and user’s experience in e- Budgeting implementation The research results propose that the role of a manager more dominantly forms users’ trust and presents important implication, and then the factor of experience in using such system and user satisfaction in operating the e- Budgeting system. 5 Sangadji (2015) To examine the main factors to encourage success in e- Budgeting implementation The results of this research show that all components of the critical success factor (CSF) model in e-Budgeting implementation with the Administration of Surabaya City may encourage successful e-Budgeting implementation. 6 Kunstelj and Vintar (2004) To identify aspects influencing e-Budgeting application in achieving its strategies objectives using some approaches. This research employs approach matrix in e- government development, and the results show that most of approaches used in e- government development have limited focus. 7 Yildiz (2007) To analyze the development and concept of e- Government This research proposes that it lacks e-government development oriented studies. 8 Evans and Yen (2006) To explore the benefits of e- Government application applied in the United States and International Organizations The results of this research show that e- budgeting application generates information which may be utilized by the public in the nation. 9 Siau and Long (2006) To identify factors which support e-government development The results of this research show the success factors of e-Government development in some nations, such as income level, development level, and territory of relevant country. This research shows different level of e-government development of every nation. Rahman, Irianto and Rosidi / Evaluation of E-Budgeting Implementation in Provincial Government of DKI Jakarta… Journal of Accounting and Investment, January 2019 | 103 Table 3 Codification of Informants No. Informant Code Position 1 Mr. GS E-budgeting Developer and Consultant 2 Mr. MC Head of BPKD Special Capital Region of Jakarta Province 3 Mr. IA Head of Budget Division 4 Mr. IR Head of People’s Welfare Budget Sub-Division 5 Mrs. TS Head of Economy Budget Sub-Division 6 Mr. YS Head of Government Budget Sub-Division 7 Mr. DS BPKD Data Team 8 Mr. AY Staff of Bapedda of Development Planning and Funding Division 9 Mr. KA Staff of Bapedda of Development Planning and Funding Division 10 Mr. OA Bappeda Data Team 11 Mrs. SS Budget Supervisor Team 12 Mr. AG Budget Supervisor Team Source: BPKD and Bappeda of the Provincial Administration of Special Capital Region of Jakarta research is used to examine the existing problems, and it requires detailed and complete understanding of an occurrence in certain site. The research collected data may be in the form of written statement, oral information, and various facts related to implementation evaluation problems. In connection with this, this evaluation employs two main techniques of data collection: documentation study and interview. In this research, the documentation study is a method to collect data by learning and noting parts deemed important of various official summary either available in the research location or in other institution related to the research location. The researcher performs independent interview for data collection. The informants of this this research are selected using the snowball technique. Snowball technique is a sampling method in which the samples are obtained through a rolling process from one to other informants. This method is usually used to explain certain social or community pattern (Nurdiana, 2014). In its implementation, this technique is performed by contacting and performing independent interview with the Head of Regional Finance Management Agency (BPKD) of the Provincial Administration of Special Capital Region of Jakarta and Developer and e-Budgeting Consultant of the Provincial Administration of Special Capital Region of Jakarta. Furthermore, the author follows BPKD’s direction to interview some e-Budgeting users in the environment of the Provincial Administration of Special Capital Region of Jakarta. The informants related to the focus of this research is shown in Table 3. Data Analysis This research employs a qualitative method with a case study approach and the data analysis technique of Miles & Huberman (2009). The reason is that the data analysis activities are performed one after another, repeatedly until the research purposes are achieved. This data analysis technique consists of the following three measures: Rahman, Irianto and Rosidi / Evaluation of E-Budgeting Implementation in Provincial Government of DKI Jakarta… Journal of Accounting and Investment, January 2019 | 104 1) Data Reduction This stage starts by summarizing the data obtained through interview with informants, occurrence and situation at research location. In this stage, the researcher chooses and summarizes relevant documents (interview result, field observation and documentation. The first activity of data reduction stage is scripting interview with informants. The second activity is reading repeatedly the script, and taking notes on other sheet of paper in case something arising in mind, presenting the researcher’s temporary interpretation. 2) Data Presentation Qualitative data presentation may be performed in the form of short description, chart, relationship between categories, flowchart and table, which serve to facilitate explanation and conclusion making. In the data presentation stage, the researcher is involved in the presentation of collected and analyzed data, in which data presentation is conducted by systematically presenting information to readers. Data presentation aims at organizing and arranging the reduced data in a relationship pattern, thus it will be more easily understood. In this stage, the researcher attempts to relevantly present or arrange data by displaying the interview data of which validity is certain and making relationship and meaning of existing phenomena, thus it will be able to make a conclusion to answer the research purposes. 3) Conclusion or Data Verification The conclusion and verification process is initial, temporary conclusion which is subject to change in case no strong supporting evidences are found in the next data collection stage. Research completeness is the basis of verification before conclusion making. The data verification in this stage is made by re-viewing raw data obtained from interview, observation and documentation, comparing one datum with other datum (data triangulation) by comparing interview data with existing field data, cross-checking results between interview with one informant and with another informant. A stronger and more accountable conclusion may then be made. Data Validity To examine the validity of data and to check the consistency and validity of data, this research employs the triangulation technique. According to Moleong (2014), triangulation is a technique to examine data validity by utilizing something else to compare the results of interview with research object. The triangulation technique used in this research is a triangulation technique with different source, by comparing a person’s condition and perspective with other various opinions and views. Interview is conducted again with different interviewees of e-Budgeting implementation with the Provincial Administration of Special Capital Region of Jakarta. Rahman, Irianto and Rosidi / Evaluation of E-Budgeting Implementation in Provincial Government of DKI Jakarta… Journal of Accounting and Investment, January 2019 | 105 Result and Discussion Budgeting of the Provincial Administration of Special Capital Region of Jakarta Before using e-Budgeting, the budgeting system of the Provincial Administration of Special Capital Region of Jakarta basically uses two different systems, held by different Working Units with their respective authorities and duties. The two systems, which are the Planning Information System (SIP) and the Regional Finance Management Information System (SIPKD), transform to the e-Budgeting system. SIP is under duty and responsibility of the Regional Development Planning Agency (bappeda) while the SIPKD system is one duty and responsibility of the Regional Finance Management Agency (BPKD). Both systems are related one another. SIP is used in initial budget planning, while SIPKD system is used to detail the budget planning. Below is the illustration of differences between e- Budgeting and previous systems (Figure 1). SIP consists of some integrated subsystems, including the Regional Medium- Term Development Plan Information System (SI-RPJMD), annual Planning Information System (SIP), Monitoring and Evaluation Information System (SI- Monev) and Development Planning Discussion Information System (SI- Musrenbang). SI-RPJMD is used to input development planning to the Special Capital Region of Jakarta Province for 5 years or throughout the leadership of elected Governor. This SI-RPJMD is also synchronized with strategic planning (Renstra) of each working unit. SI-Musrenbnbg is initial activity planning before entry to annual SIP, used to absorb public aspiration directly and in stages, starting from Neighborhood Council (RW) level. Annual SIP is used to input the development plan of the Special Capital Region of Jakarta Province on an annual basis. This annual SIP must be based on medium-term development planning of the SI-RPJMD and the Regional Government Work Plan (RKPD) of targeted budget year. This annual SIP is inputted by each Work Unit from village to provincial level with the Regional Development Planning Agency as the verifier and coordinator. Figure 1 Before and After e-Budgeting Source: www.Jakarta.go.id (2017) Rahman, Irianto and Rosidi / Evaluation of E-Budgeting Implementation in Provincial Government of DKI Jakarta… Journal of Accounting and Investment, January 2019 | 106 SI-Musrenbang and annual SIP are the means to input development planning with different method, in which SI-Musrenbang uses the Bottom-Up method to collect people’s proposal for activities, while SIP uses the Top-Down method with activities proposed by the executive. However, SI-Musrenbang remains inputted to annual SIP for verification by related technical work unit under coordination of the Regional Development Planning Agency. SI-Monev is used to monitor and evaluate the implementation of current year activities based on what is designated by Regional Government and Regional House of People’s Representatives (DPRD), which is then stipulated with Regional Regulation. Simply, budgeting process input starts from SI-RPJMD, SI- Musrenbang, annual SIP, SIPKD System and, finally, SI-Monev. SIPKD experiences massive evolution within the last 2 years, in which the Information System is replaced by the e-Budgeting system. The e-Budgeting system starts to be applied in the midst of 2013 with preparation taking only 1semester, which is the second semester of 2013. The e-Budgeting system is socialized in November 2013 to 700 Work Units with the Provincial Administration of Special Capital Region of Jakarta, performed one after another in arrangement of Regional Budget of budget Year 2014. Diagrammatically, the planning and budgeting process flow from KUA to Regional Budget may be examined in figure 2. Evaluation Results of e-Budgeting Implementation with the Provincial Administration of Special Capital Region of Jakarta 1) Context Evaluation The context evaluation results show that e-Budgeting implementation with the Provincial Administration of Special Capital Region of Jakarta has reached its objective, which is to realize the effectiveness, efficiency, accountability and transparency of planning and budgeting process in arranging Regional Budget (ABPD) and improving previous system, which is the Regional Finance Management Information System (SIPKD). Similarly, as proposed by a Staff of the Planning, Funding and Development Division (PPP) of Bappeda and the Chairman of the Budgeting Division of Regional Finance Management Agency (BPKD) as follows: Figure 2 Planning and Budgeting Process Source: www.Jakarta.go.id (2017) Rahman, Irianto and Rosidi / Evaluation of E-Budgeting Implementation in Provincial Government of DKI Jakarta… Journal of Accounting and Investment, January 2019 | 107 “The advantages of e-Budgeting implementation include efficiency in budget utilization and prevention of fraud. The efficiency is perceived by every region that they, both northern and southern, have the same unit price. Basically, the e-budgeting helps discipline the budget procedure, realize efficiency and prevent fraud”. (Mr. KA, Staff of Bappeda of P3 Division) “The purpose is transparency in regional financial management. There is difference between previous system and the e-budgeting in regard to the components. These components are the description of activities to be performed, used to organize budget with clear usage”. (Mr. IA, Head of Budget Division (BPKD)) In the context evaluation stage, besides purpose which constitutes the main focus, which is the appropriateness of e-Budgeting implementation to the policies issued by the central and regional governments, the results show that e-Budgeting application is pursuant to Presidential Instruction No. 3 year 2005 on the national policy and strategy of e-Government development and Government Regulation No. 58 year 2005 and Regulation of Minister of Home Affairs (Pemendagri) No. 13 year 2006 on the financial management of more advanced region, and Governor Regulation of the Special Capital Region of Jakarta Province Number 145 year 2013 on the drafting of regional budget (ABPD) or Amendment to ABPD through electronic budgeting (e- Budgeting). This is as stated by the Head of the Regional Finance Management Agency (BPKD) and the Head of Budget Division of Regional Finance Management Agency (BPKD), that: “The utilization of e-Budgeting is under Regulation of Minister of Home Affairs (Pemendagri) No. 13 Year 2006, as amended with Regulation of Minister of Home Affairs (Pemendagri) No. 59 Year 2007 and, finally, Regulation of Minister of Home Affairs (Pemendagri) No. 21 Year 2011”. (Mr. MC, Head of BPKD) “All systems built by the Provincial Administration of Special Capital Region of Jakarta must be based on legal basis, such as Law number 17 year 2003, Law number 1 year 2004, Government Regulation number 58 year 2005, Regulation of Minister of Home Affairs number 13 year 2006, Regional Regulation number 14 year 2011, and Governor Regulation (Pergub) number 142. Thus, everything used for e-budgeting is under legal basis”. (Mr. IA, Head of Budget Division, BPKD) On the other hand, the result of context evaluation is the adjustment of old system to new system, as stated by a Staff of Bappeda of Planning, Funding and Development Division (PPP) as follows: “It is in fact not a problem, but more precisely an adjustment, that Bappeda initially has its own system, which is e-Planning and BPKD with its SIPKD. The two systems are now integrated into one application, the e-Budgeting, for an integrated planning and budgeting”. (Mr. KA, Staff of Bappeda, P3 Division) In addition, the product evaluation shows that a change to the paradigm of e-Budgeting users from the old to the new system, which is e-Budgeting, is necessary. Similarly, as stated by the Head of Budget Division (BPKD) of the Special Capital Region of Jakarta Province, that: Rahman, Irianto and Rosidi / Evaluation of E-Budgeting Implementation in Provincial Government of DKI Jakarta… Journal of Accounting and Investment, January 2019 | 108 “The problem is that it is difficult to change people’s old paradigm or mindset to a new, better one, such as, it is initially not transparent, and so forth. But, it depends on individual and institution. However, we had relatively good ability and great will at the time, and we trained all SKPDs, or about 720 SKPDs, in the Provincial Administration of Special Capital Region of Jakarta within three months”. (Mr. IA, Head of Budget Division, BPKD). The problem stated by Mr. Iwan Angkasa is related to behavioral interest in using the e-Budgeting application. According to Davis & Venkatesh (2000), behavioral interest is a good prediction of technology utilization by system users. A decision taken by an individual to accept an information technology system is a conscious action, which may be explained and predicted according to his behavioral interest. Individual acceptance of an information technology system is determined by utilization perception and utilization easiness perception. Users will have interest in using technology when it is useful, helpful and easy to use the technology system. 2) Input Evaluation Input evaluation shows that the e-Budgeting implementation with the Provincial Administration of Special Capital Region of Jakarta is appropriate from the aspects of information technology infrastructure and human infrastructure. From the perspective of information technology infrastructure, the availability of facilities and infrastructures used, both Hardware and Software, are sufficient. However, it requires some improvement to the server used, since unpredictable overload may occur when it is simultaneously used by TAPD, and thus becomes an inhibiting factor in e-Budgeting implementation. The explanation above is supported by the Budget Supervisor Team of BPKD of the Provincial Administration of Special Capital Region of Jakarta, as follows: “There is no problem with devices like PCs, but the server must be alright to encounter any events that all SKPD systems are simultaneously used. In such a time, many SKPDs and UKPDs complain. In fact, specific schedules are allocated for them, but they input it at the end of time, which thus results in overload”. (Mr. AG, Budget Supervisor Team, BPKD) “It is actually sufficient, but the server is sometimes unable to process because of simultaneous opening throughout Jakarta, thus loading takes too long. The server and the network should be upgraded”. (Mrs. SS, Budget Supervisor Team, BPKD) The next one, input evaluation results from the aspect of human infrastructure. The Provincial Administration of Special Capital Region of Jakarta has trained the e-Budgeting users, consisting of the Regional Development Planning Agency (Bappeda), the Regional Finance Management Agency (BKPD) and SKPDs/UKPDs. The training and socialization are given in stages because of the great number of Local Government Budget Teams with the Provincial Administration of Special Capital Region of Jakarta, particularly SKPDs/UKPDs. The same is stated by the Head of People’s Welfare Budget Sub-Division (Kesra), Data Team of the Regional Development Planning Agency (Bappeda), and Data Team of the Regional Finance Management Agency (BKPD): Rahman, Irianto and Rosidi / Evaluation of E-Budgeting Implementation in Provincial Government of DKI Jakarta… Journal of Accounting and Investment, January 2019 | 109 “We initially organized internal training and socialization in one place for the staffs here, which was then extended to other SKPDs/UKPDs in stage. That the Provincial Administration of Special Capital Region of Jakarta has hundreds SKPDs/UKPDs, the training is organized several times in stages”. (Mr. IR, Head of People’s Welfare Budget Sub-Division) “In e-Budgeting early implementation, all SKPDs/UKPDs were invited in stages to show, train and socialize to them about the e-Budgeting. Any change in the future will only be explanation, in case something is missing. We made a discussion group in a chatting application. Therefore, any question of the e-Budgeting will be directly shared there, and we may also share knowledge”. (Mr. OA, Data Team, Bappeda) “Training and socialization is combined in one bundle. Therefore, in case of new feature, the Bappeda will always invite all representatives of SKPDs/UKPDs assigned with inputting duty, like head of SKPD or the operator in stages, not simultaneously”. (Mr. DS, Tim Data BPKD) Interview with some informants above reveals that the role of training and socialization significantly influences the continuation of e-Budgeting system operation. One factor of successful policy implementation, in this case e- Budgeting, is the preparedness of individuals or users and of technology and information infrastructures which play the important role. E-budgeting development must be viewed from two perspectives: internal and external. The internal perspective emphasizes more on the development of information technology infrastructure and human infrastructure, and the external perspective emphasizes on the society (Kunstelj & Vintar, 2004). The results of this input evaluation also reveal that the Provincial Administration of Special Capital Region of Jakarta always performs socialization with regard to new features and innovations in improvement of the e-Budgeting system. The innovation is not always developed by consultant or developer, but sometimes made by users when they find something needed in the field. 3) Process Evaluation The process evaluation results show that, technically, the e-Budgeting implementation with the Provincial Administration of Special Capital Region of Jakarta has been operating since 2013 and keeps developing and innovating pursuant to policy and issues arising, which eventually provide benefits until now. Process evaluation reveals that the e-Budgeting application to regional financial management provides positive impacts, since the e-Budgeting helps and facilitates Local Government Budget Team arrange the Activity and Budget Plan (RKA) of SKPD/UKPD and minimize potential errors in APBD arrangement process. For the Regional Finance Management Agency (BPKD) and the Regional Development Planning Agency (Bappeda), the e-Budgeting is quite helpful, among other, in planning process evaluation of every SKPD/UKPD, and closes any budget which is potentially inappropriate to the plan and keeps data security. The same is stated by Staffs of Bappeda of Planning, Funding and Development Division (PPP) and the Data Team of the Regional Development Planning Agency (Bappeda) as follows: “Since e-Budgeting application, the effect is mostly perceived with Regional Budget arrangement, that it is more transparent when arranged together. Rahman, Irianto and Rosidi / Evaluation of E-Budgeting Implementation in Provincial Government of DKI Jakarta… Journal of Accounting and Investment, January 2019 | 110 Thus, all people are encouraged to be transparent, that the system records the user name. With e-Budgeting, regional financial management, particularly Regional Budget is more structured and transparent”. (Mr. AY, Staff of Bappeda, P3 Division) “Since e-Budgeting is a system, there is no problem with this system that it quickly adapts to rapid change and follows it. Moreover, it has good security, that the history data is well arranged with good security data and secured data storage duplication. This is important, that a backup is ready when something happens”. (Mr. DS, Data Team, BPKD) The Provincial Administration of Special Capital Region of Jakarta always monitors and evaluates the e-Budgeting system, since this is deemed important for further system development. The process evaluation results show some constraints which must be taken care of in the implementation stage, such as adaptation to the e-Budgeting application, since this e- Budgeting is initially used by Surabaya City with about 72 SKPDs, and then used at provincial level with about 740 SKPDs/UKPDs, thus more adaptation is needed. As proposed by the Developer and consultant of e-Budgeting of the Provincial Administration of Special Capital Region of Jakarta: “Yes, the constraint is with adaptation of application or system, which was previously applied in Surabaya and then in Jakarta, which may not run as smooth as that in Surabaya. Jakarta has higher volume, and its people pay high attention and have thick political interest. The problem is more complex, thus we must adapt the difficulties operationally”. (Mr. GS, Developer and Consultant of e-Budgeting) Subsequently, the constraint of no monitoring system. This monitoring is performed by guiding regional agencies in budget arranging until activity implementation. The monitoring result is appropriateness of proposed budget to field realization. What is meant by realization is (physical) work implemented with (financial) absorption. This constraint is expressed by the Head of Sub-Division of Government Budget of the Regional Finance Management Agency (BPKD) as follows: “Current constraint is no existing monitoring, thus we cannot observe any monitoring result. This must be developed more, and monitoring must be conducted. Monitoring is important to observe how much amount is realized. If expenditure may be viewed, the remainder will also be clear. But the remainder is unclear now”. (Mr. YS, Head of Sub-Division of Government Budget) Another matter to pay attention to from the process evaluation findings is preparation of Standard Operating Procedure (SOP). The Provincial Administration of Special Capital Region of Jakarta in e-Budgeting implementation still refers to Governor Regulation (Pergub) No. 145 on distribution of duties and authorities, for SOP legally enacted which is still at drafting stage. 4) Product Evaluation The results of product or output evaluation reveal that the e-Budgeting implementation with the Provincial Administration of Special Capital Region of Jakarta helps in arranging Activity and Budget Plan (RKA) with accountable Rahman, Irianto and Rosidi / Evaluation of E-Budgeting Implementation in Provincial Government of DKI Jakarta… Journal of Accounting and Investment, January 2019 | 111 values and components. The same is expressed by the Head of Sub-Division of Economy Budget, BPKD, the Provincial Administration of Special Capital Region of Jakarta, as follows: “What e-Budgeting achieves is, first, transparency, in which the public may view everything and what the content looks like. In addition, any person who inputs will be visible, since it is recorded in history. With the transparency, anyone will be visible, including those who cheat. Finally, the stages are clear. The phases start from arrangement, discussion, and so forth. The movement will be visible”. (Mrs. TS, Head of Sub-Division of Economy Budget) The other product evaluation result shows that using the e-Budgeting, budgeting process takes shorter and on time, improves government agencies’ performance in regional budget drafting and improves budget transparency, accountability, effectiveness and efficiency. The same is expressed by Evans & Yen (2006), that the existence of a system in government environment (e-Budgeting) provides positive impact, in which the activities are more transparent and accountable, and improves the government’s efficiency and functions, particularly in planning and budgeting. In addition, using e-Budgeting improves the quality of Regional Budget from the perspective of appropriateness to planning document of RPJMD (Regional Medium-Term Development Plan). Disclosure of output evaluation results may be viewed from two perspectives: effectiveness and efficiency. The effectiveness achieved by the e-Budgeting is the appropriateness of RKA to designated RPJMD since it contains budget allocation of activities and standard price usage. Using standard unit price, it is expected that there will be no difference in component price between regions. The efficiency may be viewed from timeliness of RKA arrangement, as expressed by Mr. Gagat Sidi Wahono as the Developer and Consultant of e-Budgeting, as follows: “Operationally, APBD is on time, printed pursuant to the designated format and the data are secured. E-Budgeting may change the mindset and working pattern of governmental agencies in the Special Capital Region of Jakarta. The phases between planning and budgeting stages are planned better. Furthermore, the responsibility and easiness of tracking from a data or from an activity at any time, or, in other words, the budgeting control management runs”. (Mr. GS, Developer and Consultant of e-Budgeting) In addition, this stage reveals that with e-Budgeting, distribution of duties and authorities are clear and budget control management play an active role. Conclusion Based on the results and discussion in previous section, we may conclude that the e-Budgeting implementation with the Provincial Administration of Special Capital Region of Jakarta is worthy to be continued since it evidently runs effectively from the perspective of CIPP model (Context, Input, Process and Product). The first component, context, focuses on the successful achievement of initial purpose, which is realization of budget transparency and accountability with the Provincial Administration of Special Capital Rahman, Irianto and Rosidi / Evaluation of E-Budgeting Implementation in Provincial Government of DKI Jakarta… Journal of Accounting and Investment, January 2019 | 112 Region of Jakarta. The second component, input, reveals that utilization information technology infrastructure in the form of existing facilities and infrastructures is good and adequate but has weakness with the server used. The utilization of human resources or human infrastructure is sufficient since in its implementation, the Provincial Administration of Special Capital Region of Jakarta always performs training and socialization with all Local Government Budget Teams (TAPDs). The third component of CIPP model, process, discloses that by using e- Budgeting application or system, TAPD more easily help arrange RKA (Budget Activity Plan), the positive impact of this process evaluation result minimizes errors in inputting RKA arrangement process and distribution of duties and authorities is clearer. In e-Budgeting implementation stage, an SOP (Standard Operating Procedure) needs to be arranged immediately. Finally, from the perspective of product component, the evaluation result states that the existing e-Budgeting implementation with the Provincial Administration of Special Capital Region of Jakarta improves the quality of APBD from the perspective of appropriateness to planning document of RPJMD (Regional Medium-Term Development Plan). In addition, the advantage of e-Budgeting is that the budget control management plays an active role. The weakness of this research is that it has only few informants, since in order to improve the effectiveness of research time, the researcher focuses more on the internal factor of Regional Finance Management Agency (BPKD) and Regional Development Planning Agency (Bappeda) of the Special Capital Region of Jakarta Province. This research should add more informants from the Regional House of People’ Representatives (DPRD). Further researches are expected to employ informants from the Regional House of People’ Representatives (DPRD) in order to enrich information of the effectiveness of e-Budgeting implementation in regional planning and budgeting process. References Anwar, K., Djumiarti, T., & Rihandoyo. (2013). Analisis Faktor-faktor keberhasilan Elektronik Governemnet pada Pemerintah Kabupaten Jepara. Journal of Public and Management, 2(4), 87-96. Retrieved from https://ejournal3.undip.ac.id/index.php/jppmr/article/viewFile/3587/3 498 Arikunto, S. (2014). Penelitian Tindakan Kelas. Jakarta: PT. Bumi Aksara. Chen, Y. C., & Zhang, J. C. (2012). Citizen-centric e-government performance: satisfaction with e-information. Electronic Government, an International Journal, 9(4), 388. https://doi.org/10.1504/EG.2012.049726 Creswell, J. (2014). Penelitian Kualitatif & Desain Riset: Memilih di antara Lima Oendekatan (Edisi Indonesia). Yogyakarta: Pustaka Pelajar. Darodjat & Wahyudhiana, M. (2015). Model Evaluasi Program Pendidikan. Islamadina, 16(1), 1–28. Davis, F. D., & Venkatesh, V. (2000). A Theoritical Extension of The Technology Acceptance Model: Four Longitudinal Field Studies. Management Science, 48(2), 186–204. https://doi.org/10.1287/mnsc.46.2.186.11926 Dunn, W. N. (1999). Pengantar Analisis Kebijakan Publik. Yogyakarta: Gadjah Mada University Press. https://ejournal3.undip.ac.id/index.php/jppmr/article/viewFile/3587/3498 https://ejournal3.undip.ac.id/index.php/jppmr/article/viewFile/3587/3498 https://doi.org/10.1504/EG.2012.049726 https://doi.org/10.1287/mnsc.46.2.186.11926 Rahman, Irianto and Rosidi / Evaluation of E-Budgeting Implementation in Provincial Government of DKI Jakarta… Journal of Accounting and Investment, January 2019 | 113 Dwidjowito, R. N. (2004). Kebijakan Publik Formulasi, Implementasi dan Evaluasi. Jakarta: PT. Elex media Koputindo Kelompok Gramedia. Evans, D., & Yen, D. C. (2006). E-Government: Evolving relationship of citizens and government, domestic, and international development. Government Information Quarterly, 23(2), 207–235. https://doi.org/10.1016/j.giq.2005.11.004 Farsi, M., & Sharif, M. (2014). Stufflebeam ’ S Cipp Model & Program Theory : A Systematic Review. International Journal of Language Learning and Applied Linguistics World (IJLLALW), 6, 400–406. Hakan, K., & Seval, F. (2011). CIPP evaluation model scale: Development, reliability and validity. Procedia-Social and Behavioral Sciences, 15, 592–599. https://doi.org/10.1016/j.sbspro.2011.03.146 Hasan, A., Yasin, S. N. T. M., & Yunus, M. F. M. (2015). A Conceptual Framework for Mechatronics Curriculum Using Stufflebeam CIPP Evaluation Model. Procedia-Social and Behavioral Sciences, 195, 844-849. https://doi.org/10.1016/j.sbspro.2015.06.324 Jakarta, B. P. K. D. P. D. (2017). e-Budgeting Aplikasi Perencanaan dan Penganggaran (Pendekatan Berbasis Program, kebutuhan dan Harga Satuan). Retrieved from http://akuntansidki.com Jones, S., Irani, Z., & Sharif, A. (2007). E-government evaluation: Reflections on three organisational case studies. Proceedings of the Annual Hawaii International Conference on System Sciences, 1–8. https://doi.org/10.1109/HICSS.2007.189 Kementerian Koordinator Bidang Perekonomian Republik Indonesia. (2015). Tinjauan Ekonomi dan Keuangan (Mendorong Pembangunan Infrastruktur Demi Pemerataan Kesejahteraan Masyarakat). Tinjauan Ekonomi dan Keuangan, 5(4), 6–7. Kunstelj, M., & Vintar, M. (2004). Evaluating the progress of e-government development: A critical analysis. Information Polity: The International Journal of Government & Democracy in the Information Age, 9(3,4), 131–148. Retrieved from http://proquest.umi.com/pqdweb?did=881207981&Fmt=7&clientId=8 991&RQT=309&VName=PQD Mahmudi, I. (2011). CIPP: Suatu Model Evaluasi Program Pendidikan. Jurnal At-Ta’dib, 6(1), 112–124. Miles, M. B., & Huberman, A. M. (2009). Analisis Data Kualitatif. Jakarta: Universitas Indonesia Press. Mohebbi, N., Akhlaghi, F., Yarmohammadian, M. H., & Khoshgam, M. (2011). Application of CIPP model for evaluating the medical records education course at master of science level at Iranian medical sciences universities. Procedia-Social and Behavioral Sciences, 15, 3286-3290. https://doi.org/10.1016/j.sbspro.2011.04.287 Moleong, L. J. (2014). Metode Penelitian Kualitatif. Bandung: PT. Remaja Rosdakarya. Nugroho, R. (2006). Kebijakan publik untuk Negara-negara Berkembang. Jakarta: PT. Elex Media Kompetindo kelompok Gramedia. Nurdiana, N. (2014). Teknik Sampling Snowball dalam Penelitian Lapangan. Contech, 5(2), 1110–1118. Prasetyo, D. H. (2006). Penerapan E-Budgeting pada Proses Pembuatan rencana Anggaran belanja Daerah Kota Surabaya. Paper was Presented at Seminar Nasional Manajemen Teknologi III. Surabaya. Purwohandoko, Sanaji, & Mustofa, A. (2015). The Successful Implementation of E-Budgeting In Public University: A Study at Individual Level. Journal of Advances in Information Technology, 6(3), 135–139. https://doi.org/10.12720/jait.6.3.135-139 Quezada, G. V. (2005). Performance Evaluation Models For Strategic Decision-Making: Towards AHybrid Model. Michigan: Faculty of The https://doi.org/10.1016/j.giq.2005.11.004 https://doi.org/10.1016/j.sbspro.2011.03.146 https://doi.org/10.1016/j.sbspro.2015.06.324 http://akuntansidki.com/ https://doi.org/10.1109/HICSS.2007.189 http://proquest.umi.com/pqdweb?did=881207981&Fmt=7&clientId=8991&RQT=309&VName=PQD http://proquest.umi.com/pqdweb?did=881207981&Fmt=7&clientId=8991&RQT=309&VName=PQD https://doi.org/10.1016/j.sbspro.2011.04.287 https://doi.org/10.12720/jait.6.3.135-139 Rahman, Irianto and Rosidi / Evaluation of E-Budgeting Implementation in Provincial Government of DKI Jakarta… Journal of Accounting and Investment, January 2019 | 114 Graduate Collafe in Partial fullfillment of the requirements for the Degree of Doctor of Philosophy Departement of Educational Studies Western Micgigan University. Retrieved from https://scholarworks.wmich.edu/cgi/viewcontent.cgi?referer=https://s cholar.google.co.id/&httpsredir=1&article=2054&context=dissertations Sangadji, E. F. (2015). Faktor-faktor yang Mendorong Keberhasilan Implementasi E- Budgeting pada Oemerintah Kota Surabaya. Thesis, Universitas Brawijaya, Indonesia. Sari, K. D. A., & Winarno, W. A. (2012). Implementasi E-Government System Dalam Upaya Peningkatan Clean and Good Governance di Indonesia. Jurnal Ekonomi Akuntansi dan Manajemen (JEAM), 11(1), 42–54. Sesmiarni, Z. (2014). Model Evaluasi Program Pembelajaran. Bandar Lampung: Aura Printing & Publishing. Siau, K., & Long, Y. (2006). Using Social Development Lenses to Understand E-Government Development. Journal of Global Information Management, 14(1), 47–62. https://doi.org/10.4018/jgim.2006010103 Stair, R., & Reynolds, G. (2010). Principle of information System (9th ed.). USA: Course Technology. Stufflebeam, D. L. (2003). The CIPP Model for Evaluation. Dorderecht: Kluwer Academic Publisher. U. N. E.G. (2016). E-Government in Support of Sustainable Development. Weber, R. (1999). Information System Control and Audit. USA: Prentice Hall Inc. Yahaya, A. H. (2001). The Using of model Context, Input, Process and products (CIPP) Learning Programs Assesment. Research Report, Universiti Teknologi Malaysia, Malaysia. Yildiz, M. (2007). E-government research: Reviewing the literature, limitations, and ways forward. Government Information Quarterly, 24(3), 646–665. https://doi.org/10.1016/j.giq.2007.01.002 https://scholarworks.wmich.edu/cgi/viewcontent.cgi?referer=https://scholar.google.co.id/&httpsredir=1&article=2054&context=dissertations https://scholarworks.wmich.edu/cgi/viewcontent.cgi?referer=https://scholar.google.co.id/&httpsredir=1&article=2054&context=dissertations https://doi.org/10.4018/jgim.2006010103 https://doi.org/10.1016/j.giq.2007.01.002