Developing Web-based Education Resources: Lessons Learned From Three Experiences By JoAnn Carr, University of Wisconsin-Madison Abstract This article reviews the development of three Web-based education resources and the potential for each of these resources to meet the needs of users for a 'killer app'. Three case studies (the Annotated List of Education Journals, the IDEAS Portal Web Site and the Eisenhower National Clearinghouse Web Site) review the purpose, audience, content, funding, publicity and structure of the sites. Differences in staffing, funding and the centrality of these sites to the mission of their sponsoring institutions impacted the growth of these sites. Technological changes and the diffuse nature of the Internet also impacted the development of these resources. Introduction "The education killer app is coming'' declared Levinson and Suratt in Converge (2000, p. 7 4 ). Among the rea­ sons for their assertion is vendor development of "fully integrated Web-based accountability solutions" (2001, p. 74) . However, this author's experience in developing Internet resources in education at the local , state and na­ tional levels belies the implication that the education 'killer app' will be a comprehensive commercial product. Rather than a 'prie fixe' served by a single provider, I believe the decentralized nature of the Internet will provide a smor­ gasbord of possibilities from which educators can select the precise menu items that meet individual needs and tastes to create a personalized 'killer app'. Each of the case studies resources shows the potential for each resource to users needs to be met in developing his or her personalized 'killer app' . The three case study sites serve as subject gateways to educational resources in that they are "selected and evaluated by subject experts or in­ formation professionals" (Pitschmann, 2001, p.4). They provide added value to selected content through de­ scription, resource organization and collection mainte­ nance. Two sets of factors are examined for the impact on the development of these sites: 1.) the differences in staff­ ing, funding and the centrality of these sites to the mission of their sponsoring institutions, and 2.) the technological changes and the diffuse nature of the Internet. Education Libraries Volume 25, No. 2 Winter 2000 Case Study One: A Local Project The Annotated List of Education Journals (ALEJ) is an example of a project that began as an individual initiative, progressed to a departmental initiative and is now transformed into a managed collaborative project. Jeneen LaSee-Willemssen began the list in the spring of 1996 as a resource to provide clients of the University of Wisconsin-Madison Instructional Ma­ terials Center (IMC) with more detailed information about the journal titles held by the IMC. The list in­ cluded a description of each title with links to the pub­ lisher, sponsoring agency and indexes in which the title was included. The value of this resource was not limited to its intended local audience. Before the IMC could announce this resource to the local audience the site was chosen as a Yahoo "Pick of the Week" in May 1996. The ability of a resource to be used be­ yond its intended audience is an integral part of Internet publishing and drove subsequent decisions about de­ velopment of this resource.The evolving focus of ALEJ reflects its growth in use from a local to a national resource. During the summer of 1996 decisions re­ garding the scope and structure of this resource were still being made. In September 1996 the scope of ALEJ was defined as "education titles currently held by the IMC". Library science titles were to be included only if they were indexed in an education database. Children's magazines held by the IMC were included with the intention to split them off into a separate list in the future ).(Personal communication, October 14, 1996). 13 The audience and content of ALEJ changed during the fall of 1996 when Lasee-Willemssen began at new po­ sition at Kansas State University (KSU). During that time the list became a joint venture of the IMC and KSU . The criteria for inclusion of titles expanded beyond IMC holdings to include "other journals that support K-12 education and teacher education" (Cohen, 1997a, p. 5) . Among the titles listed and annotated were those included in major educational indexes, Internet re­ sources and children's magazines that support the K- 12 collection . New to ALEJ were electronic journals in education that did not have a print counterpart. These titles were fully annotated with direct links to the online text. Other links provided in each entry included tables of contents, manuscript guidelines and educational as­ sociation home pages. The list continued to be a valuable Internet resource . ALEJ was the most frequently visited section of the IMC site and was linked from other sites at the Universities of Chicago, Kentucky and Yale as well as from the Los Angeles Unified School District. In addition it was cited in guides to education research as an "informative, an­ notated list of 426 current education journals" (Pea, 1999, p. 338). However, no specific funding was com­ mitted to this site and publicity was sporadic, relying on word of mouth and serendipity rather than planned and systematic resource announcements. This lack of fund­ ing and sustained publicity arose from the fact that the list was not central to the goals and mission of either the IMC or KSU and therefore did not meet criteria neces­ sary for sustainability. It was time to move this project to a managed collaboration . In the fall of 1999 Renee Sengele, a student in the School of Library and Information Studies at UW Madison, ex­ pressed interest in completing a practicum in Collection Management at the CIMC (the Center had changed its name and structure in the fall of 1998). In consultation with KSU and CIMC staff Sengele developed a process to move ALEJ from a list to a database format. This evolution required review and revision of this resource, now defined as: "to provide K-12 professional staff and teacher education faculty with a searchable database of relevant English language print and electronic jour­ nals focused on K-12 education and professional staff preparation" (Carr, 2000a, p. 1 ). Annotations, Internet links and the assignment of sub­ ject headings and educational levels would assist the target audience in identifying professional titles of in­ terest to them. Subject headings and educational levels were chosen from a standardized vocabulary based on the Thesaurus of ERIC Descriptors. Education Libraries Volume 25, No. 2 Winter 2002 The entries in the current list were reviewed to deter­ mine appropriate fields including field labels and types. Each record in the database would be constructed as illustrated in Figure 1. Determination of the appropriate database structure rep­ resented a substantial step towards a managed collabo­ ration. Discussions between the author and LaSee­ Willemssen in the summer of 1999 had focused on the need to improve the sustainability of the resource by providing for shared development by education infor­ mation professionals and other members of the target audience. Therefore, the decision was made to de­ velop a Web accessible database using a standard bib­ liographic tool that would support a self-submit form for additional entries to the database. This decision guided the work of Sengele and necessitated the development of a proposal to the Digital Project Steering Committee of the University of Wisconsin Madison Libraries . The March 2000 proposal was accepted and the initial plan­ ning meeting with technical staff was held in September 2000. The meeting focused on determining the standard bib­ liographic tool to be used for the database. Site Search from OCLC was chosen because it would provide local control, flexibility and shorter implementation time. Un­ fortunately the lack of centrality of this project to any of the collaborators negated the short implementation time. Neither the CIMC, Lasee-Willemssen (now at the Uni­ versity of Wisconsin-Superior) nor the Library Tech­ nology Group of the UW-Madison Libraries were able to focus exclusively or even primarily on this project. The Annotated Database of Education Journals was released and announced to the target audience in early 2002. A full five years after the list was begun as a locally focused individual project, this resource is on the verge of serving as a part of the 'killer app' smor­ gasbord as a broadly focused, decentralized, managed collaboration. Members of the target audience and edu­ cation information professionals will be able to contrib­ ute to the development of this resource so that it better meets individual needs as well as those of other K-12 teachers and teacher educators. The Annotated Edu­ cation Journals Database demonstrates the evolution of information providers from serving local audiences to groups defined neither by time nor location but by information need. ---==== Education Journals : An Annotated Database for K-12 and Teacher Educators Available at: http://webcat.library. wisc.edu:31 021 14 - Case Study Two: A Statewide Project In contrast to the Annotated Database of Education Journals, the IDEAS Portal Web Site was envisioned from the beginning as a collaborative project. The au­ thor was invited to an initial meeting to discuss the de­ velopment of the IDEAS portal on May 25, 1999 and to share her earlier work in the EdLibWeb Project. 1 Rep­ resentatives from fourteen higher education and K-12 agencies met in response to concerns regarding the need for: • integration of technology into the curriculum, • access to selected, tested Internet curriculum materials, • coordination and communication among prof­ essional development providers, • and greater interaction among teachers . (personal communication, May 7, 1999) Discussions at this meeting resulted in the decision to meet these goals through the development of a por­ tal web site that connected Internet resources to the Wisconsin model academic standards. Core values in the development of this site were quality control, teacher review and testing and ongoing assessment of the site. A second meeting of what was now called the IDEAS Advisory Group was scheduled for August 18, 1998. In the interim, small groups met to determine the ba­ sic organizational structure, scope, quality indicators and the roles of IDEAS coalition partners. At the meet­ ±:g tiE AdviD:ryG DJP GgJeErl th:tttre IDEAS Portal Web Site would begin with a database of Internet re­ sources focused on the needs of Wisconsin teachers, providing links to materials useful to Wisconsin class­ rooms that was organized by subject areas and the GEM standards and could be accessed with a power­ ful search engine. In October the vision of the IDEAS Advisory Group was presented at the Governor's Wisconsin Educational Technology Conference. This vision was articulated as: "to create a portal web site that will provide se­ lected, high-quality curricular and professional devel­ opment resources for educators having a user and con­ tent focus" . The goals of the site were to be interac­ tive, to provide high quality, to be supported by the for PK-12 community, to engage teachers and to address [Wisconsin] State academic standards and mandates (Carr, Jacques & Trick, 1999).1n subsequent months core members of the IDEAS Advisory Group, serving as the IDEAS Executive Group, developed a work plan, timeline and budget launching the IDEAS Portal Web Site. The operational model for the site would include an advisory coalition to provide general oversight and a full time project director to manage the site. Education Libraries Volume 25, No. 2 Winter 2002 The content of the site would be developed by groups of subject specialists from PK-12 and higher education institutions. A Web Manager would provide the technical development for the site. Initial site develop ment would focus on identifying resources in Mathema tics, Language Arts/Reading, Social Studies and Science. During the summer of 2000 members of the Executive Group hired a doctoral student in Education, Tom Pedroni, to develop a prototype database that would serve as a proof of the concept to potential funding agencies. In consultation with the Executive Group, this graduate student developed a quality rubric [See Figure 2] Using this rubric Pedroni selected eighty resources that reflected topics relevant to the fourth, eighth and twelfth grades and supported specific standards in Mathemat­ ics, English/Language Arts, Science and Social Stud­ ies. Each entry included the title, URL and author of the resource as well as an annotation, subject head­ ings, and links to relevant Wisconsin Model Academic Standards [see Figure 3] . This selection of a strategically distributed group of re­ sources coupled with a structured vocabulary and search engine ensured that relevant subject area resources would be found as the prototype database was demon­ strated to teachers and to potential funding agencies. A presentation of the prototype at the 2000 Governors' Wisconsin Educational Technology Conference drew an audience of over 100 individuals in contrast to an audience of less than 30 at the previous year's confer­ ence. During the fall of 2000 the site was also pre­ sented to the TEACH Committee on Collaboration. This Committee is comprised of six Wisconsin state agen­ cies and associations concerned with PK12-higher edu­ cation in Wisconsin. 1 As a result of this presentation and of the interest expressed in the project by Wiscon­ sin educators, TEACH-Wisconsin (Technology for Edu­ cational Achievement in Wisconsin), the University of Wisconsin System and the University of Wisconsin­ Extension agreed to provide financial and in-kind sup­ port for the development of the IDEAS Portal Web Site. The first IDEAS project director was hired in February 2001 and plans to launch the Web site by August 2001 were developed. The project director reviewed andre­ vised the quality rubric, worked with the Web Manager to revise the site and move it from Access to Oracle, and recruited, trained and supervised the first teams of researchers. The teams of researchers were comprised of faculty from K-12 and higher education with exper­ tise in specific subject areas. These researchers se­ lected, evaluated, annotated, tested and developed da­ tabase entries for quality Internet resources. ' See Carr. J.A. (April 13. 1997) .EdLibWeb: A Study in Cooperative Web Site Devel­ opment. Paper presented at The Association or College and Research Libraries Conference, Nashville, TN. 16 - Figure 2: IDEAS Quality Rubric Title: Reviewer's Record Number: Reviewer: Rating: 2=Excellent 1=Good O=Poor NA = Not Applicable Format __ User Friendly Aesthetically Courteous (downloading time is reasonable, easy to read, standard viewing for various settings and browsers __ Aesthetically Appealing (attractive design and coloring) Format Total Content __ Credible (information is accurate, complete, and routinely maintained) Useful __ Information is rich and likely to be revisited __ · _ Interdisciplinary __ Demonstrates sound theory of multicultural education Avoids gender, race, culture, economic, religious, and other biases that im pede inclusion of all learners Appropriate to grade level __ Incorporates State and/or National Standards or Guidelines Content Total Education Libraries Volume 25, No. 2 Winter 2002 17 Figure 3: Sample IDEAS Record Title: Anti-Imperialism and The White Man's Burden http://tlc.ai.org/lessons/antiimpe.htm Subject: Level One: Social Studies Level Two: U.S. History, World History Keywords: Imperialism, anti-imperialism, the white man's burden Description: In this lesson, students explain what was meant by the phrase "White Man's Burden" and evaluate differing opinions on the role of the U.S. in the world prior to WWI. After discussing the pros and cons of an imperialistic foreign policy at the turn of the 19th century, students are asked to draw parallels between the anti-imperialist debate of the 1890's and the debate over America's role as the "world's policeman" in the 1990's. Creator: David Rodgers, Zionsville Community High School Standards: Scheme: Wisconsin Model Academic Standards Grade: 12 Topic: Social Studies Standards: Social Studies 8.12.1 Explain different points of view on the same historical event, using data gathered from variou~ sources, such as letters, journals, diaries, newspapers, government documents, and speeches. 8.12.2 Analyze primary and secondary sources related to a historical question to evaluate their relevance, make comparisons, integrate new information with prior knowledge, and come to a rea­ soned conclusion. 8.12.4 Assess the validity of different interpretations of significant historical events. 8.12.5 Gather various types of historical evidence, including visual and quantitative data, to analyze issues of freedom and equality, liberty and order, region and nation, individual and community, law and conscience, diversity and civic duty; form a reasoned conclusion in the light of other possible conclusions; and develop a coherent argument in the light of other possible arguments. 8.12.7 Identify major works of art and literature produced in the United States and elsewhere in the world and explains how they reflect the era in which they were created . 8.12.15 Identify a historical or contemporary event in which a person was forced to take an ethical position, such as a decision to go to war, the impeachment of a president, or a presidential pardon, and explain the issues involved. Resource Type: Lesson Plan Grade Levels: 10-11 Education Libraries Volume 25, No. 2 Winter 2002 -----------------------------=--~~~--------------- 18 The vision of the IDEAS project as a collaborative activ­ ity is reflected in the nature of its funding as well as in the diversity of its advisory board. The core values of quality, teacher review and testing and ongoing assess­ ment envisioned for the site at the initial planning meet­ ing still held true. Quality control is supported by the use of a revised and expanded version of the initial rubric, the use of multiple evaluations, and a selection rate of 17% (IDEAS, 2001, p. 4 ). Teacher review is ensured not only by the involvement of classroom teachers and teacher educators in select­ ing the sites but also by the commitment of the IDEAS researchers to test the selected resources in a class­ room setting. The commitment to ongoing assessment is provided through the opportunity for feedback by the users of the site and well as by formal evaluations. As noted in Bright IDEAS for Wisconsin, " ... the IDEAS project is always a work in progress. We will continu­ ally edit and update the resources, our processes and the web site to meet the demands and needs of Wisconsin's educators" (IDEAS, 2001, p. 11 ). The IDEAS project has demonstrated its potential to be part of the education 'killer app' by a well defined pur­ pose and audience, by providing for quality of its con­ tent, by developing a structure by which this resource would be the central purpose of its founding coalition, by providing for continual promotion, and by focusing on meeting the needs that had led to the initial meeting of the IDEAS Advisory Group . The identification of the specific Wisconsin academic standards supported by each resource was an integral part of the information provided by each researcher. The project manager also continued to promote and publicize the site including meetings with members of the IDEAS Advisory Coalition and presentations at state conferences. On July 25, 2001 the IDEAS Advisory Group met to review the status of this two-year old project that was officially launched on August 1, 2001. Case Study Three: A National Project Unlike the first two case studies, the third entree in our education 'killer app' smorgasbord is neither an indi­ vidual nor collaborative project. Rather the Eisenhower National Clearinghouse (ENC ) has been funded by the U.S. Department of Education since 1992 "to support improvement in mathematics and science education throughout the country" (Mathematics & Science Edu­ cation, 1997, p. 2}. Education Libraries Vol.25 No.2 Winter 2002 The role of the Eisenhower National Clearinghouse is : • to collect and describe K-12 mathematics and science education materials including print, audio, video, multimedia, kits and Internet sites. • to disseminate information about the collec­ tion through ENC Online (available via Internet or a toll-free number), CO-ROMs and print materials. • to collaborate with the Regional Consortia to create demonstration sites throughout the nation (Mathematics & Science Education, 1997, p. 2). The development of the ENC site stands in contrast with the other two sites described in this article because its purpose, audience, content and funding were defined from the outset of this project. The central funding for ENC provided for the recruitment and hiring of staff which would support the operational structure of this resource. In addition to staff to manage the technical infrastructure and subject area experts in science and mathematics to review and annotate the resources, other staff was hired for outreach, library and information systems, acquisitions, and evaluation.Aithough the or­ ganizational structure of ENC has changed over the years, these core areas of service have remained. For example, the ENC staff was reorganized into teams in 1996 in order to provide for the publishing teams (print, CD-ROM and online) to work on integrated products . The development of the ENC site stands in contrast with the other two sites described in this article because its purpose, audience, content and funding were defined from the outset of this project. The central funding for ENC provided for the recruitment and hiring of staff which would support the operational structure of this resource. In addition to staff to manage the technical infrastruc­ ture and subject area experts in science and mathemat­ ics to review and annotate the resources, other staff was hired for outreach, library and information sys­ tems, acquisitions, and evaluation. For example, the ENC staff was reorganized into teams in 1996 in order to provide for the publishing teams ( print, CD-ROM and online) to work on integrated products.( print,CD-ROM and online) to work on integrated products. Although the organizational structure of ENC has changed over the years, these core areas of service have remained.ln April1993, representatives from 19 twenty-four ENC constituency ing the cataloging of re­ sources, technology delivery systems and standards for acquisition of materials .The initial focus of the Clearing­ house was defined as providing a catalog of primarily print materials, but the discussion did include a recom­ mendation that "ENC should use the Universal Resource Locator to identify where materials may reside in data­ bases throughout the Internet" (ENC, 1993c, p. 7). The Collaboration Workshop included a demonstration of WAIS and Gopher technologies, prompting the obser­ vation that although the technology to access the Internet is not in the schools now, ENC must keep abreast of changes in technology. (ENG, 1993c, p.18).The con­ tent of the database as selective or comprehensive was also debated with the conclusion being to develop a com­ prehensive resource that provides "sufficient informa­ tion about materials so that users can make intelligent choices on their own" (ENG, 1993c, p. 1 0). At a Standards Task Force meeting in September 1993, the need to consult teachers in developing the vocabu­ lary for the project and in designing the search inter­ face was stressed. The Task Force also recommended that ENC have a paper submission form for materials as "people are afraid of machines" and "the cost of elec­ tronic submission may be prohibitive" (ENG, 1993b, p. 6).The audience for the second Collaboration Workshop in April 1994 demonstrated the interest of ENG in meeting the needs of a diverse audience as sixty individuals attended this work­ shop compared to twenty-four at the 1993 workshop. The im­ pact of changes in technology from the previous year was re­ flected in the focus of the workshop on opportunities, "including ways in which ENG could make full-text of curriculum materi­ als available online" (ENC, 1994b. p. 2). The continued emphasis on meeting the needs of its audience and ENC's role as part of a larger initiative was indicated by reports from eight focus groups comprised of teachers from eight of the ten regional consortia. Among the many accom­ plishments of ENC in that first year was the establishment of an Internet site with gopher and World Wide Web access. This site was demonstrated to the Mathematics and Science Advi­ sory Boards in October 1994 where Board members were in­ troduced to the ENC graphical client that would allow users to access images directly from the ENC catalog. The Third Col­ laboration Workshop was held in May 1996with seventy par­ ticipants. In contrast to the first two Collaboration Workshops, this event looked more at the implications of use of Internet resources such as ENC, rather than at the development of the resource. Lessons Learned Although each resource grew out of a different need, varied in the size and focus of its audience, differed in content and had As other developers work to meet the Education Libraries Vol.25 No.2 Winter 2002 needs of educators and add to the rich menu for the "killer app'. The following attributes need to be a part of the develop The purpose, audience and content will deter­ mine the role and vision for the resource . That vision requires a champion committed to sustaining the vision and to meeting the identified needs: Meeting that need requires that the developer responds to changing con­ ditions in funding, in technology or in extension of audi­ ence and purpose. The developer must be willing to seize opportunities that provide for a refinement of the resource. Most of all, the resource must become central to the mission and vision of the sponsoring agency, for "ultimately it is not the practice itself but how well that practice lends itself to a particular set of goals, workflows and staffing pref­ erences that determines its effectiveness and value in any given setting" (Pitschmann, 2001, p. 2). References Annual report of the Eisenhower Mathematics and Science Consortia and Clearinghouse (1998). Washington, D.C.: U.S. Department of Education, Office of Edu cational Research and Improvement. Carr, J. A. , Jacques, M. & Trick, N. (1999) . 1DEAS: web basedcurriculum resources.Paper presented at the Governors Wisconsin Educational Technology Conference: Milwaukee, Wisconsin; October 14, 1999. Carr, J. A. (2000a).Proposal for the Digital Project Steering Committee.Unpublished manuscript. Carr, J. A. & Deau, A. (2000b}. Using Internet resources for standards-based education.Paper presented at the Governor's Wisconsin EducationaiTechnology Con ference, Madison, WI : October10, 2000. Cohen, M. L. (1997). Education journal annotations online. IMC Newsletter, 7, 3. Cohen, M.L. (1997a, spring). Education journal annotations. IMC Newsletter, 7, 5. Cohen, M.L. (1997b, summer) Journal access improve­ ments. fMC Newsletter, 7, 5. Educational Links.(2001 ). Los Angeles Unified School District. Retrieved July 13, 2001 from http://www.lausd.k12.ca.us/orgsNision_ 4/links. 20 Eisenhower National Clearinghouse for Mathematics and Sci­ ence Education . (1993a).Mathematics and Science Advisory Boards Meeting 2: September30, 1993. Eisenhower National Clearinghouse for Mathematics and Science Education .Report of the National Network of Eisenhower Mathematics and Science Regional Consortia and National Clearinghouse. (1997). Columbus, OH:Eisenhower National Clearing­ house. Eisenhower National Clearinghouse for Mathematics and Sci­ ence Education. (1993b) .Standards Task Force Meet ing: Summary Meeting: September 2,1993. Eisenhower National Clearinghouse for Mathematics and Sci­ ence Education. (1993c).Summary and Minutes: Col laboration Workshop : April28- 29, 1993. Eisenhower National Clearinghouse for Mathematics and Sci­ ence Education . (1993d).Summary: Mathematics and Science Advisory Boards Joint Meeting: June 10, 1993. Eisenhower National Clearinghouse for Mathematics and Science Education.(1994a).Mathematics and Science Advisory Boards Summary: October 4-5, 1994. Eisenhower National Clearinghouse for Mathematics and Science Education. (1994b).Second Annual Col/abo ration Workshop Summary: April28, 1994. Eisenhower National Clearinghouse .Mathematics and Science Advisory Boards Meeting.(1995). November30- December 1,1995. Eisenhower National Clearinghouse . Mathematics and Sci ence Advisory Boards Meeting Summary. (1996) . October 31- November 1, 1996. Electronic Journals at University of California.( 2001 ). Retrieved July 13, 2001 from http://www.lib.uchicago.edu/e/su/ edu/elists.html. IDEAS Portal Web Site. (2001). Bright IDEAS for Wisconsin . [Brochure] Madison, WI: University of Wisconsin Board of Regents . Education Libraries Vol.25 No.2 Winter 2002 IDEAS Portal Web Site :Pre-proposal draft. (2000) . Unpublished document. Levinson, E. & Surrat, J. (2000, May) . The education killer app is coming. Converge, 5, 75. Monk, J. S. (1996). ENG collection development policy. Columbus,OH: Eisenhower National Clearing house. Pea, R. D. (1999). New media communication forums for improving education research and practice. In E. J. Lagemann & L. S. Shulman (Eds.). Issues in educa tion research: problems and possibilities. San Fran cisco: Jossey-Bass. Pitschmann, L.A. (2001 ). Building sustainable collections of free third-party Web resources. Washington, D.C.: Council on Library and Information Resources, Digi tal Library Federation. Simutis, L. (1994). The Eisenhower National Clearinghouse for Mathematics and Science Education: working toward education goai4.Educational Media and Technology Yearbook, 20, 120-125. World Wide Web Subject Catalog.( 2001 ).Retrieved on July 13,2001 from http://www.uky.edu/Subject/educationall .html Yale University Social Science Libraries and Information Services Subject Guide: Education . Retrieved 2001 from http://www.library.yale.edu/socsci/subjguides/ education/education.html JoAnn Carr is the Director, CIMC, University of Wisconsin, Madison, WI. E-mail: carr@education.wisc.edu 21